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ABSTR-ACT 

« . Beginninjg* vitlr the* planning stages of the^Hational 

AssessMent.o^ Educational Progress (HAEP) , careful attention has been* 
given to the 'design of efficient probability .sampling Methods for the 
selection of class-age respondents and the asslgntent of test* 
package?, l^ith th^se Methods, it is pbssib^e for VAEP researchers tt> 
■a Ice relatively precise statSMents about population characberistics 
on t!^e basis of fairly SMall saMples. The purpose of t^is Myograph 
is to describe what is Meant by relatively precise st'ateMents about, 
population .characteristics and to shov hov NAEP saaple data are being' 
tts6d to gauge the accuracy of -repotted ites^lts. Tiie levels of 
precision for fear 01, and Q2 vere coMpared, and the overall precision , 
was iMproved in* Tear 02. The s&Mpling error Methodology *devel.(^ed j^or 
the tear 02 saMple was applied 'to calculate Tear 03 and 04 saMpling 
errors. A Major redesign of> 9ABp*s^.priMary saMple vas initiated for 
the Tear 06 assessMent. The T«ar (!« ("T97i»-75) HASP in-schopl .priMary 
saMple vas an independent replicate of the tear 05 saMple selected 
frOM the deeply stratified priiary unit fra«e developed for the* 
1973-74 survey. Four ' non-overlapping saMples vere to^ be used 
successively for Tears 07 through 10. PriMary t^j^s of inforMatton 
provided by report: Procedures (SaMpling) (Evaluation) « (BR) ' 
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ter 1: ll^TRODUCTION 



;?ational As sessiaent* Overview - ^ - ' 

The ilational Assessment oI^Educatlonal Progress (KAEP) can be viewed 
as an annual series of large-^cale sample surveys designed to iseasure the 
educational achievexaents of four age groups in 10 subject areas* The four 
specific age groups include all 9-year-olds and 13-year-olds enrolled in 
school at the tisiA' of assessment and all H^-year-olds and young adult^, 
.ages 26-35* The first assessment of science , writing, and citizenship 
spanned the 1969-70 school year* Subsequent'^ assessments have been conducted 
during the 1970-71, 1^71-72, and 1972-73 school yea^s* At this writing', 
the Year 03 assessment of Career and Occupational Development (COD) and 
Writing is xmden/ay, and' planning for tlie Year 06 assessment has beg\in. 

V National Assessment r^espondent$ answer questions and perform tasks 
much the same as they would on a typical f achievement 'tfiest. One aspect of 
Nationife Assessment that ^d^^tinguishes |t from the typical educational 
testing program is thfe way data are irenotted. Instead of calculating test 
scores for each respondent and forming^^tiormative distributions*, i^esultd on 
each released exercise a?:e reported s4^arately* Unreleased exercises are 
held' back for reassessment in subseqj^nt years so that trend measurements 
will not be biased by school systems teaching to specific NAEP exercises « 

u 

The reporting of separate exerciseat, takes the form of estimated proportions 
Responding correctly, ^within variou^' subgroups x>f the^target population* 
Group effects that ^contrast the proportion of correct ansvets for a specific, 
subgroup against the corresponding national proportion are used to detect 

« • • • * N 

variations in knowledge, understand^ing, skills, and attitudes among various 
segments 0^ the population* With this method ''of reporting, it is not 
/necessary for each respon4dnt to com;Dlete the entire set of exercises. 
Subsets of exercises, called packages, are formed,' which take approxicbtely 
50 minutes each tp complete. 10 such packages are formed for a particular 
age^class assessment, then 10 nonoverlappihg samples, each representative 
of the target population^ ar^^specified and assigned a par tlcular. package. 

Beginning with the early planning stages of National Assessment, careful 
attention has bden given to the design kid implementation of efficient 



probability sampling oeChods for the selection of age^-class respondents 

and the assignment of packages.^ With the^i:iethods. It Is p<>sslble for 

IIAEP researchers to make relatively precise stateinents about ^relevant popu^ 

*| » 
lotion characteristics on the basis of fairly scall sauries v The purpose 

_j)f this mnograph la to describe what Is d^^t by relatively precise state- 
ments about population characteristics and to show how Itatlonal Assessmei^t 
sample da^xa are being use^ to gauge the accuracy ^f reported results. 

Population Characteristics and Sanple Statistics ' v 

' Whll^ statisticians and other researchers familiar tylth survey methods 
are well avara of the inferential ''leap*' that Is made when sample-based 
results are taken to represent population facts, many users of sample ^ata 
do not readily distinguish be^een population parameters and sample statistics 
'it is the researcher *s obligation, therefore, to polnV out that his survey 
results are an Imperfect approxlmatloh of the truth, an approximation whose 
accuracy is X?.mited by his financial resources and His ^sample survey skills. \ 
The sources of er^r that plague survey results are numerous. Many of 



these ejror sources— suCh as wiuseable responses to vague or sensitive 
questions; no response from particular sample members; and errors in coding, 
scoring, and processing the data — are beyond the control of the sampling 
statistician. The nonsam{)llng errors are also common to complete enumerations 
of a target population^ such as \he U.S. Decimal Census. One advantage of a 
small sample survey over a complete enumeration, In addition to ^he obvious 
cost savj^gs, is that a smaller, ;^re highly trained, and supervised field 
force followed up by careful scoring and processing, of' the small sample data 
may produce fewer npnsas^ling errors per respondent than the *large unwieldy 
census operation* » 

*In addition to poor response, nonresponse, scoring, and processing 
errors, sample survey results are innaccurate ffredsely because they are based 
on a sample and not on the. entire population* Consider^ for exampl^e^ the 
population percentage of. <9-yeat'-olds who cartf answer a particular science 
exercise correctly. For^a specl£ll&d s^unple design and selection procedure, 
a very large number of possible^samples could bc^reallzed. Supose that 
s 1^^,... S indexes the totality^ of possibly samples that could be dra^ 
in accordance With a sjrt^cified procedure. A' probability sampling method is 



distinguished by fact that each sanple-s has a known nonzero prftbability 

or being s&lec'ted. If we denote this probability of se"lect}.on by "-(s) and 

" let P(s) denote the sample-s estimate for the percentage of 9-year*olds who 

can answer correctly, then * 

^' S ^ ji . - 

■ ^ . ^ ; E{P(s)} - *E ;t(s) P(s> ' . (l.JL) 

, ^ • . .3*1 . 

^ f ' ^ P • ' ^ K 

is the expectation, or expected value, "of the sample sf^tistic P(s)\ ' This 

expectation represents the average value of the estimates ?(s) oyer a con* 
, ceptually infinite sequence of repeated sample drdws with t:{s) denoting , 
the frequency , of occurrence for samRle-s7~~ If this expected value does Tiot 
equal the population parameter of interest, say then P(s) is said to be 
a biased estimate 6f P. The magnitude of this bias is specified by , 
* * ' * ■ . - 

Bias {P(s)} - [E{?(s)} - PJ. ■ \ (1.2) 



Bias in a sample statistic may be attributed to nansampling as well as * ^ 
sampling sourcefe^^^th£t ±s, statistics that would otherwise average oat 
to the true population yalVie can miss the mark -if nonresponse, i;ieasureaent , 
or .processing errors are m^de. In the ^absence of ^nonsampling errors, 
probability samples provide^f^ unbiased estimation of population totals < # 
like the numerators and denominators of NAEP P-values^ Or^the otner^hand, « 
strictly unTilased estimates for ratios of population totals are often uaavMl- ' 
able. The sampling *6lases assoCiate4 with, ratio estimates are generallv 
negligible when large-scale probability samples are involved. Some empirical 1^ 
evidence for this contention is presented in chanter 4, Hhere the. sampling 
biases of NAEP P-val,ueg are studied.. ' ... ^ 

Besides the systematic errors that cause, the sample estimate to miss 
the mark on the average, one must also recognize that it is possible tio ^ . 

hit the target on the average wfitle missing .the. bull's-eyS^^bstdntially ^ ' s. 
in some ^sataples, Jo quantify these random sampling fluctuations, , 
statisticians, have defined the sampling vaj^iance of P(s) as 

' Var (ks)) - S'7i(s) [P(s) -S{P(s)}J^ ' ' (1.3)- ' * 

8-1 
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TKIs quantity represents the squared distante between the sample- values 
and their expectation or centroid averaged over an infinite sequence of 
sain^;Le draws. A aore appropriate neasure of sample dispersion for a, 
^biased estimator is the mean squared e r r o"rr~^~w4 igh t^ed average of squared ^ 

differences between sample values and ,lhe true populatio;i value P: - 
"* \ \ S ' ' ' " * 

'\' ^ MSE |p(a)} - I :i(s) [P(s) -.PJ^ . 

" . a«l " 

- \ . • . ■ , ' . • . * 

The mean squared error of a sample statistic has an obvious relationship * 

to l^s bias and variance; namely, < ' ' * 

- \ ■ . " . ^ . , • " 

\ MSE {P(6)} - Bias^ {P(s)} + Var {P(s)} / ' (1,5) 

The quantity most comoply "use^* to characterize.' the sampling variation of 
a statistic is called the standard ejrror or SE{P(s)}, where 

SE {P(s)} - [Var {P(s)}]V2 ; ' (l.g)' 

\ An analogous, quantity for biased statistics is 

• TE {P(s)} - [MSE'{P(a))]^/^ '■ ""(l.?) 

often called the "total error" or root mean squared error. 

It is aftjiarent fron the definitions in ^uations 1.3 through 1.7 that 
the true value of these saiapUn^ erroj^ measures -Cannot be <letemined from 
a single sample, ft is possible, however, to produce valid escinates of 
these quantities using the data obtained frcto a w*ell-designed probability 
, saiflple. Probability samples which provide lor estimating the sampling ' ^ 
^variability, ordinarily the standard errors, of sample stati-stics have been 
called lacasurab-le [ref, ]]. E:eanples of nonmeasurable probability samples 
include systematic random selections from lists exhibiting periodicity and 
^stratified random samples with a single unit selected* p"er stratum. 
\^ National Assessment ^s committed to the desigii of measurable samples, 
s^le^" which provide for reasonably valid estimates of stlflS-ard errors. 
Thyse standard errors, ua'ed 1ft , connection with respected statistical con- 
ventioijs; .make it possible to bridge the gap between' sample estimates and 
population facts-.. A statistical framework for inferring population P-values 
and f^r inferring gxoup feffec^s fioq saml,le effects "la outlined in the - 
<follo;^lng section. . ' ' 

s 



Statistical Jni^erence - , ^ 

Cbfifldenoe Intervals , ^ ^ ^ ^ ^ * ♦ ' - * ' 

• Khen one isakes' inference froD a sanple^ aoout the magnitude of a popu- # 
lation paraneter, like.-P, by quoting a sample estimator ?(s), it is common 
«t^ti^tical practice to include a range or interval of^values about P(s) 
which fte likely tto contain the true population value P. * Such interv;als 
are coraonly called 'Confidence intervals" in the statistical literature; 
they'^requcntly take the form * • ■ ' ' 

Ip(sy - P(s) + k se.{p(s)} . . ^ * (1\8) 

wherp is a constant and se f?(s)} is the estimated st>andard error for 
the sample statistic P(s).^ The "confidence coefficient" associated with 
such an interval is the probability that a randomly selected sample will 
yield an interval Ij^(s^ thiat includes the true population value P, 
Recalling that we have S possible samples which ape realized^ith prob- 
abilities tt(s), this confidence coefficient can be specified by defining 
A(P) a^ the set of samples where the interval Ip(s) contains ? and letting ^ 

' . y(P) - ■ Z :r(s) (1,9) 

• seA(P) 

s 

denote the probability that thS^lnterval associated -with a randomly selected 
sample will tontain P. Notice that the summation in equation 1.9 extends 
over all samples-s' which belong to thfe set A(P) [3£A(P) denotes s belonging 
to A(P)J. In empirical terms, this probability statement means that, in 
a conceptually infinite sequence of repeated san^le draws, a f raction y (P) * 
of the corresponding intervals will contain P. 

' T ♦ 

In order to specify 4 value of k in equation 1.8 that will yield'^an 
internal with given confidence coefficient y(P)» one must know the samRling 
distribution of the standardized variable ' - 

' ^ e'Cs) - [P(s) - Pl/se *{P(s)} - ^ * " (1.10). 

Notice* that the set A<P) of samples with;Ip(8)EP [Ip(s) containing PJ is 
equivalent -to the set/of samples with !t(s)l<k. It is clear that the *^ 
sampling distribution of t(9) cannot be specified exactly without- a "complete 
enuascration of the target population. To: pursue this line of inference, 



sampling statisticians copmonly assini^ that the sactpling distribution of 
t(s) can be approximated by Student's T distribution .with (df) ^degrees of 
freedom 'or bjr the standard normal distribution when df e:cceeds 60. Thfe 
rationale for "tfilCarassumption rests on the "tendency of statistics iike P(s) 
4rom large probability samples to have nomal-jLike sampling distributions, 
^Wtth P(s) ai)proxlmately normal, the sampling. distribution of t(s) vill 
resemble StudentJs T with^ the appropriate degrees of freedom. 

For a" stratified multistage saiiqile^ with a total of n primary san^ling 
units (PSU&) selected from H primary strata, the degrees of freedom 
associated with t(s) can be approrLnated by df - (n-H). Some authors ^ave 
reconaended a more sophisticated approximation for df attributed to 
Satterthwaite Iref. 2]. Satterthwaite's appro:?imation attempts to account for, 
unequal within-stratum variance comi)onents an<i„;sja|ying sttatum sample siies. 
The results of some recent empirical studies summarized in chapter A of this 

• monograph seem to "inditfaoe that 'the naive Approximation for df, namely 

• df - (n-H), is to be preferred. ' 

A further characterization of a y(P) confidence iAtfervil can be made 
in terms 6f its so-called Operating Characteristic (OC) curve. ♦ OC 
curve summarizes the probabilities that ptAnts P* other than the true value 
P will be included in th'p interval- corresponding to a randomly selected 
sample. If j^e let y(P*) - Pr {Ip(s)cP*} where l^{s) has the form in 
. equation -i. 8 then ' . • ^ - 

^ ' - Pr {U(s,A*)| < k} ' . (1.11) 

;^ere , • * . ^ ' , 

t(s,A*) lP(8) -.P*l/s6 {P(s)} 
^ t(s) + (P'-P*)/se {P(s; 
- ,t(s) + i*/se {P(s)} 

J»fs the form of Student's noncentral T statistic fiith df degrees of freedom 
and noncMtrality t)a^amete:; 6* - A*/SE{P(|^}T Fcjr values of P* deviating . 
considerably from the true value P, one would hope that y(P*) would be small. 

It is important to note at <:his ppint that, -for a giveh .saipple design 
and an estimation scheme characterized by SE{p'(s).}, and .the degrees of 
freedom-df asaociatecTwith ;se{P(s) } , the entire OC curve is specified once 



k is set* With this in oind,. iris 'clear from equation- 1.11 that, while 
an incteastf in k'will raise the confidence level, the associated 

Interval,^ it will" also » inflate the p^rob^ility of including unwanted values^ 
Another way of viewipg 'this relationship between increasing conf idence**and 
the iiiclu^ion of taore unwantjed values (?* i P) is gained by observing that 
the expected lengph of a randoQ interval such as IpCs) in equation 1.8 is 
direotly proportional to k. Hence, the greater'the confidence, coefficient 
the wider the Interval. The value is most cocsionly set to- yield 

confidence coefficients in th^ neighborhood of .95 of '.99. • ^ 

Significance tests ^ * ^ - ^ ' - 

When a sizeable group effect is observed in the sample, one can ask if 
it is likely that iuch an effect could be due solely to sampling variations. 
To answer such questions, statisticians have devised an inferential structure 
ki^own as the test of significance* Ve^wiU describe this structure in the 
► context of :iational Assessment "grouo effects": 

' J' ' . ^ . . ' \ 

. ^PtvCs)-* [Pq(s) - P(s)l • . (1.12) 

where rPgCs) denotes the saaplfe-s est4.1aate.of the proportion of group G 
«iibers who can answer ' a particular exercise correctly and P(s) depicts 
the corresponding proportion for the entire population. Group G could, 
for example, denote the 9ryear-olds residing in NAEF's Northeast region,-, 
in which case iP^Cs) would compare the performance of the ^Northeast 
9-year-olds against the' overall national peaepmanc€£ of 9-year-olds. i 

An. observed group effect APg-Cs) is judged to be significantly different " 
from zero if its absolute value exceeds a, critical v^lue C- The critical 
value is determined so that the probability of observing an Absolute effect 
APg(s) ln_ excess of C^vAen the true population effect AP^ is zero is less < 
than some a^rbittarily small probability cr. This probability a. of declaring 
an" observed sample effect significant vhen in-iact the true population effect 
is zero^ called the slgnificanct level of the test. Commoniy used ■ 
significance levels are 0 - .01 arid o " «05. The> critical value C frequens^y 



talces the foro 



C p(s) - k se • (1.13) 



Where k is a constant and serAP(s^} is the estimated standard' error for the 
group effect AP(s). The subscript^9 deaigndting a particular subgroup, has 
*been dropped from the group effect *'s^ol in equation 1.13 to simplify our ^ 
notation. It we let A(iP) denote ^he set of samples for which iP<s) exceeds 
k se{a^(s)} in absolute value and use*^ , 

, ^ ci(iP) ^ I tt(s) . (1.14) ^ 

scA(iP) 

to denote the probability* that an observed group effect AP(5) will be 
judged significant, then a(AP) can be expressed as follows: 

. a<AP)''- Pr{|t(s,iP)|>k} . ' .(l.iS)' 

where • , 

t(s,iP) • AP(s)/se{iP(s)} * ' 
/ '^ - [aP(s)*- iP]/se{iPU)} + APV3e{AP(s)} 'f 

- PCs)- + iP/se{iP(s)>. ^ - ■ ■ . 

Notice that, as with the OC curve presented in equation 1.11 for otif confidence' 
interval, cs(fiP) can 'be specified in terms of the. sampling distribution -of a 
statistic t(s,AP) whi^h has the form of Student^ s noncentral T statistic. 
'If aP, ,the true population group effect, were zero, then^a(O) - ?r{|t(s)J> k} 
represents the significance level of the test with t(s) talcing the fom of 
Student's central'! statistic. For populations with AP»«0, a<AP) gives the 
probability of declaring significance when the true yroup effect is iP. *Taken 
as a function of iiP;, the curve a(i?) described in equation 1.15 is called • , 

. the power function of Ae signific^ce test. As AP deviates increasingly 
from zero, one would .hope y:hat a<iP) , the probability of declarfhg signifi- 
cance, would 5ise sharply. . • . * ' ' ' . • 

While .the OC curve- for our confidence interval could be completely 
detemined if the populatioj^ was fully specified, only one point, of the"* 
power curve can be dttermined: namely ^ that point corresponding to the true 
'group effept iP°. The, other poiijts are conceptual in th^i^sense that they 
specify what- the probability of declaring significante would be fot a Similar 

/•population wher^ the true group effect -Was iPVfiP°. 



12 



8 V. 



p 



?-rescril>iftg a critical. valu6 for a test of significance that will yield 



r a predfetferaiined significatice level » liresumes. knowledge of the sampling 




intervals,^ sampling statisticiane cora?oniy asSune that Student's centjral 
J distribution j?puJl,d^ be a reasonable approximation for the sampling dist^-i. 
butiprf of t(s',^) 'from a population with - 0. If* the d^rees'of ire^don ^ 
'aa8oci5te,d with*se{AP(s)} ^exceeds 60,. one can effectively use the standar.d 
normal, distribution to defcemliie k such^hat Pr{J"'t(s,0) i> k} - .a. Tj-pical .. , 

values of k'^rom the Stand^d normal distrfb.ution .are k « 1.96 for a 
significance level 3 -".05 and k - 2.58 for a significance level. of kOi". 
E::amining^the ,fora of the "powey function" in equation 1.15 makes it clear 
t^at*. vJhile ope ^ reduce the risk of falsely declaring sigiviiicance "(that 
^ \ is reduce a) by increasing k, there will^lie a corresponding- reduction in 
the power to declare significance when t(le true group effect A? deviates 
from zero. This same relation*iip wasVdced between increasing confidence I 
coeffi-cients and lengthening intervals/ . - " ■ \ 

• nn addition- to the direct comparisons betw^rv. subgroup "and national ■ I 

proportions of correct answers which we have called group effects, National 
Assessment rep.p.rts adjusted or balanc6d effects which attempt to correct 
^for Che masquerading, of xjne characteristi-c-as the effect of another. WhlTe ■ ^ 
the unadj^usted group effects properly raflept the dif feren(;e*s in achievement, . 
between specific groups of children, much of - the observed' dif Arence nav" 
well b'e attributable to otljer factors ^on which the- , compared groups differ. 
For 'example, part qfrtifw* deficit in 'aS^eWnt observed in tl\e direct . 
comparison of Black, students with non-B}acTis may be attribufed" to^the fact ^ 
that Black students tend, more than non-Black students, to haVe less 

educated parents. In the foiloying sfection, the addustment methoaoiogy " * ■ 
used by Kati^^al Assetfsient to compensate for sgme oT-this^masqudradlng is ' 
■presented.* . ' . . ' ^ 

* * 'I . \ 

Balanced Eff ects ■ ' • * t \ * • ■ " 

^ ' -Jhe maj^f population subgroupinTgs use.d in National Assessment reports 
' are: -Aget Region, Si^e and Type of Comunity '(STOC) , Spx, Color, and. 
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Parents* Educ6tlozf« Within tb^ four age classes, group'^eff ects contrasting 
' mk levelsL of the other five factors are presented • As we have indicated, 
these direct coinpaxrlsons across the levels of a single factor are subject 
to masquerading InflpencW^f the other four factors. ' This confusion is 
partially due to the mbalWiced nix of these other characreristics across 
' the levels of any jingle factor being examined. - To balance out this dis- 
proportionality, National Assessment forms adjusted' group effects (expressed 
In percentages) that, whe» combined by addition with each other aniJ wi.th 
the overall "national*' percentage of success, give fitted percentages of 
^success (P-values) that correspond with th^ actual sample data, in the • 
following way: ^ 

If w^cl^p toy level of^a single characteristic, say Blacks,^ 
and use ^Wl , fitted P^value an4 estimated populatio;i size to 
calculate the number of '^succeasfes for each Heglon ic STOC x SEX x 
Parents Education subclass of 'Blacks,, Md*tjien add th^se predicted ^ , 
numbers of successes^ the predicted nun^er pf successes* ove^ all * 
these subclasses will be the same as the tqtal number of Black 
successes estlmatfed from the" sample data* " ^ J 

' If we let 1-1(1)4 Inde^s NAEP/s fou^ regions; J - 1(1)/ the seven . 
STOC categories;"kVl,.2 the two sexes; A - 1(1)3 the three-color classes: 
add n - 1(1)5 IlAEP's five levels of Parenps' Education /then the fitted 
P-value for aubciaas (iJkAn) has , the form , . ' ' 

. . ^ ' '• 

PajUm) -.?-+ AR(1aM^(J) + AS(k) + LZil) + AE(m^ " (1.16) 

where P is. the overall* (ndtdonalj'percent^orrect and^e L terms represent 
,the l^alanced' group effects for Region-i, STOC-J-; Sex^, Color-i, and 
Parents' Education claas-m.' With M(ijkAm')^'li»Mting the. estimated population 
size"fOT sUbcl«8s (ijk£m) and Y(iJkAo) re'presftakng' the estimated riunSer of 
correct responses from this subclass, the balancing condition verbalized 
above translates, into the following fitting -equations: 

t I. Z Z M(ijk£m) P(iJkJlm) - Y <i-H-H-) for i - 1(1)4 /(l,I7a) 
J-1 k"l £-1 xsfl Bai. 



•Z Z t ^ M(ljk£D)XiJkim)^. Y(+J+H-) f^r J - 'l(l-)7 , (1.17b) 
4-1-^1 i-1 m-1 • ; Bar' .^^^ 



. Similar ^ets of equations arfe producei for theT other three classifications' 
.b7 siming over all the subgroups within a particular factor, level and 

^ equations to the estimated total correct for that factor level • \otice that 
we "have used a plus sign to denote a sum&d over subscript. Substituting 
the lin^dr aain effects aocel in (1.16) for P(tjk.:n) the fitting equations 
becoce: , - - 

. - ' 7 . 2 . 

M(1)P + M(l)AR(i) +. "i Hdj) iiT(j) + I M(lk) AS(k) (1.18a) • 

. ' • j-1 • k-1 ' / 

' 3 . . 5 . . . 

+ : >!(U)iC(*) + M(in)iE(tt)'- for i - 1(1)4 

£-1 ^.1 ' ' . 

and • - ' » " 



r 



.M(j)? + : :'i(ij)ii(i) :'i(j)iT(j) + : :i(jk)iS(k) (i.isb) . 

» i-i . . • k-i . . 

' 3 * . 5 ^ 

. -h- E MCu)iiCa) * : M(jayiE(n) - Y(j) -for j,- 1(1)7 • 
i-1 * " 

The^xDther* three sets of fitting equations are arrived at sisiilafly. t^otice 
tha: we have suppressed the sutaied-over subscripts to laake the expressions 
nore^ftonpact. • ' ' 

Since each of the sets of fitting equations correspondibg to a*.^particalar 
classification factor susis to the sacve quantity, naaely * 

. M ? + 2 » M(1)AR(1) t : M(j)iT(j) + S M(k)iSCk) (1.19) 
i-1 ^ , k-1 

+ M(Z)^C(il) «f r M(a)<lE(n) - V " , ' 

i-1 ffl-1 . . ■ ' / ' . 

one or the e^ations In each set is- redundant ♦ That is, of the A + 7 + 2 
3 + 5 21 balancing equations produced in this fashion, only 16 arfe independent. 
^ To solve tor our 21 balanced effects we need f Ive . ad^lrional e<luaticns. 
Requiring Chat the overall;? in our codel (cqu3:tion 1»16) be eijuivaler.t to ; 
the unadjusted natil>nal ?-value (? - 'i / :i) i=?lies*in •q;:ation 1^15 tha't ^ 



* 
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' , 'i JKiURd^J- : A(3)iT(j> r.-h^^ih^kk) (1.20) 

i-l ^, • JTl ■ • 

.-2 M(i)AC(i) - Z M(a)iEOn) (f * . ■ • 
A«l - .o-l* * , . . ' ^ 

^Setting ftach of these* suds 'eqiial to zeto yields five Independent equations, 

whi;^h c«l be ^ub^pituted VespecCivel^or the last equation in each of the 

original five seti. This yields^. 21 independent equations, which can' be 

solved to yield the lalf set .6f balanced "effects. 

'■ , . # 

, While this balancing soXw;;ion was not derived with the least squares 

principle In aind, one can \rlev( tha restiltsr as a s^la fcsciaate of the 

xeast squares solution thePt would-be obtained if the entire oot)ulatlon of 

ccrrecc-lncorrect (2>-0) respo^jl^s wey:^ predicted by a linear rodel with an 

intercept and 21 dusz^r variables indicating meab^rshi? in the 21 factor 

level subgroups. The weightetf restrictions in equation (1.20), with the 

"hats" renoved from thi population sizes KMs) » are cocaonly, applied to 

tmbalanced data sets*. This duzEay-^'Variable regression view of HASP's "balanced 

fitting plaeis the results ii^ja fanill*ar StatlAticaif setting where the 

0 aHjust^nt of regression coeffidjaAts for'mibalanced repfesentatlon across 

categories "is a well-known"?r5pert7. ^ • * 

While balancing helps, to .coyrect for disproportionate ntabers, this 

adj^ustTient is ob^Tiously 'linited- to the vatiables that are used in the ' 

analysis* Other ucneasiftred'^va'tiables such as 'faaily InccM also be 

causing isa^ueriding probj^e^s/ Some variables used in the adjustment, such 

as color, classify respondants too coarselyi while other factors, such 

as parent's ed^atioa,' give' only. an indirect indication of the'oarfeats' 

attitude toward education or fheir. inclination to, assist' the* student with 



hoaework. Another potential problen 'with. dire.fit conparisons be^c/een sub- 
groups, is the tact that 'the .'jjerforaance of a given subgroup cay differ froa 
one subgrouping to another in -the ocner variables. That is, the effects 
associated with Black s.tudejits Say be different- in the West than in the 
Southeast. Such ir^teracticiff effects are not accounted for in 1:a£? 's- balancing 
nodel, la splta'of these deficiencies, balancing represents a big sees :'ron 



the outward appearances of unadjusted group effect* toward the inward 
realties of cause and effect. * , 



1. Kish, L. (1965). Survey SanplAag . New York; John Wiley and Sons. 
^*""*^^*^"» ^* 2' (I94e>* 'An approximate distribution of 
■ r- I °f variance -conqiornents. Bioiaetrflcs- Z, 110. 



• Chapter 2t YEAR 01 SAMPLING ERRORS ^. 

Design Description • ^ ^ ^ ^ * * 

The iJAEP Year 01 sample for the three in-school age classy (9, 13, 
and 17) began with a highly stratified, siople random selection of 208 
primary units. These ^Imary units .consisted of clusters of schools^ formed 
within selected lifting units. 'Xhe listing units were counties or parta 
of counties i Variableft^ used^ to stratify these listing units included (1) 
Region (4 Geographic Regions), (2) SOC (4 ^S^ze of Community* Classes),' 
and (3) SES (2 Socio**Economic^Status Categories). Within 6ach selected^ 
listing unit a separate set of schools was selected for each of three age 
groups: 97year-olds, 13-year-olds, and 17-yeayrolds . For each of these 
age gtoups, schools were grouped such that every set would contain a mix 
of high atid low SES students.^ Portions .of some large schools were allowed 
to belong. to more than one group. The number ,ci schools' in each of these 
clusters was based on the numbers of packages or questionnaires required 
from each PSC. The 17-year-ol<i> assessment , fv example, employed 11 separate 
group-fdminist^red packages and 2 individutlly administered packages. Group 
administrations consisted of 12 sCudents, while each individual package was 
given separarely to 9 students in eaah PSU. \ / " y 

The sample was designed to yield two primary units frbm each of 104 str^- 
ta^ Fbr the 17-year old assestfment, (1^ x 12) + (2 x 9). - 150 studi^ats wete 
required from each PSU. The groups of 17-year-old schools were constructed 
to^ contiift approximately 300 l;7**y^ar-olds each. Once a cluster of schools 
^as selected via simple r^dom sampling (SRg) from tnos^ constructed^^ the 
group packages were allocated to schools. Each school in the cluster was 
assigned a number of group, administifat Ions roughly proportional to its en- ^ 
roXJLment of 17-year-olds. Sixteen students were selected for each group ses- , 
sion assigned to a j^articulj^r scho^: 12 tcf participate and 4 to be altertsates. 

The two Individual packages were allocated to schools such Chat tor 
each group package from 1 througl^ 9 .assigned to a ^thopl, an administration^ ^ 
of individual package 13 was also planned. Individual package 14 administra- 
tlons 'were similarly .linked to aa^inistratlpns of group* packages 3 through 
ll!! For each .individual package administration planned for a school, two 



students were selected, oiya to participate and one alternate. . This 
design yields a {Planned sample size of 2,4*48 students for eaclv group** 
adninlstered. package^ and 1,836 for each Individual package. 

The Year 01 out*-of-schqol saople of young adults 26-35 and out-of- 
school 17-year-olds used the' same basic primary sample design as ^the ' 
In-school sample. The same random draw was used to select TSUs '£n bp.th 
samples; however, the out-of-jschool PSUs were defln^ In terms of a set 
of area segments or. clusters containing an ^verage of 35 to 40 housing . 
units. Each of theSe PSUs was constructed so as to contain about 16,000 
persons, ^e second-stage s^ample was a stratified random cluster sample 
with two clusters selected without replacement from each of five strata. 
The stratification was based on aiv ordering of segments In terms of the 
^precent of families earning less than $3>000. The high poverty (low SES) 
quarter of the list was assigned two strata for a cwo-to*one oversampillng 
Qf the low SES quarter. Each household clu3ter was expected to yl^ld 12.5 
eligible adult respondents*. Ten packages ci exercise^ were admlnl'Stered 
to young adults with each respotldent Randomly assigned a single package. 
Out*of-school 17-*year-olds encounte«d lr\ the household sample were asked 
to respond to a set of four or five of^ the XT-^year-old ln*school packages. ' 
Recall that there were 13 such p^ackages* An Incentive payment of 10 dollars 
was given for completing 'the se^ of packages. 

ParamgtSrs of Interest 



Proportions Correct (P-Valoes) 

^he purpose of iT&tlohal ^sessment (Year QiO^was to produce baseline 
estimates of the proportions d£ potential respondents vho would answer 
certain exercise .in a particular %^ay. ^ Restricting/ our attention to a'' 
partlcultg: In-schooltage^^lfe^ (say 17-year-olds) and a particular* exercise 
within one of the packages, l^t 4 v 

s f llf the tk-th student: in. school/(J) of-PSU (1) In 



hlj^. |"stratq;a-h ^nswer^ correctly; '(J otherwise * * 

The population means of these 0, 1 variables are the populat$birp^oportlons 
Vpf Interest I that is 



H S 



hlAlj 



:• • , . .h-l/Mj-lk-l ^^^^ 

were • i \ ' 

E «.the riuaber of >scraca (X04 planned) 
S, - the number pf PSUs in stfatua (h) 
the nvsaber of schools in PSU (hi) 
Mj^j^y y the number of 'student a in school (hlj) 
and 



hU 1-1 j-1 



/ 



hlj 



Our sample escisaces for chese proportions are of« Che fors 



(2-.1) 



H TL s m , . 
P - Z Z" Z"^ Z"H W^,,^ Y^.,, /K 
. h-11-1 j-lk-l ^^^^ ^^^^ ' 

where ' ' , ) 

■ the nuaber df PSUs selected for the sample from stratum 
■ ' (h) (generally n, 2) f 

the number of schools, In^JSU (hi) in whltfji the particular 



(2.2) 



'hi 



package oz Interest was administered 



• Qj^^j * the^ number of students fro^ school (hij) who respond to the 

paclcage of interest; 
and» asid^ from nonresponse adjustment?, * 

Vjk ■ ^^^l^' ^ Pr*fSch (j)l(hi)) X Pr {Kid (k)[(Mj)} (2^3) 

wich ' V ' . < " ' ■ 

Pr {PSU (hi)} ' ' - njj/Njj - . . ' _ ^ 

- Pr-tsCH(j)|(hl-)} ^ - Tijj^j I 

. Pr (Kid (1)1 (hlj)} 'Wi^ni^iy / . . ; 
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For group- administered pacltages the number of sample schools per PSU (Sy^^) 
was always Tin Year 01. Individual packages were 'administered in' more ' 
than one school; that is, s. > 1 fUr Year 01 individua], package exercises. 

The estimation of out-of-schoo,l, young adult, P-values parallels the 
'procedurfi presentVd for the Year 01 In-school saiaple'.. If we let j subscript 
area segments instead of schools and k young adults Instead of studeijts^ the 
expressions^ in equations 2.1 and Z'Z are Interchangeable* To complete the 
^tch, we let S^^ denote the number of segmdiits In the ESU^hl frame and 
s tSve'nuaiber.of sample segments In PSU-hl (usually. s,, « 10).. Also, let , 
^Ij ^^^^^ number of eligible young adults In segment-hlj and n^^^ ^ , 
the number of young adults In segment-hlj re^ondlng to a particular package. 
The oiit-of-school sample weights reflect the selection probabilities for 
young adults plus adjustments for nonresponse. 

Out-of ^school 17*y ear-olds located and tested In the household survey 
were combined with In-school respondent^ to estimate a single P-value for 
all 17-year-olds. The total number. of out-o£*school 17-year-olds and the ^ 
number that could' respond correctly were estimated for ^ach 17-year-old 
p^ckagh^ using weight sums for all package respondent;!^ and ^f or all respondents 
anSweri-ng correctly. These estimated totals wez'e then added to the 
denominator ^and numeratjor of the ln*school package P-Valu^. ^ 

Sub^opulatl6n P-Values and AP Values 

In addition to the natriona^l' P^Values discussed in the previous section, 
, certain subpopylation breakdowns were of interest. For example, P-Valqes ^ 
have been presented by Region, STOCf Se^c,* Color, and Parents*. Education. 
These stibpopulatlon TP-Values were produced by including onlj those observations 
^ ^ 1)elpnglng to the subpopulatlon of intetWfc in the numerator and denominator: 
^ of ^qu^tloii 2.2» , Differences between subpopulatlon and national P-(Values 
wferrstujSV^d to assess the main "effects of Region,' STOC, Col%r,^Sex> and 
Parents* Education. Xh^se direct comparisons were Introduced as group effects 
or AP-values in chapter 1. ' , • , 

Balanced Effects - ^ - , * j ' 

In chapOe^ 1 Mt lntj:oduced NA£P*6 algorithm £or adjusting aroup effects. 
This adjustment was designed to correct for the masquerading effect o( A* 
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ancillary variables when their distributims vary acros^ the levels of the 
^factor being examined, the adjustment or balancing algorithm used amounts 
to a set or linear equations which can be viewed as a sample approximation 
to the normal equations that would result from a ^JLeast-squares fit to, the 
population of 1-0 (correct-incorrect) responses based on an intercept and 
duzmy variables indicating the levels' of NAI?*s five reporting categories. 
A set of restrictions are imposed on the balanced effects » which force / 
thfe linear model intercept to equal the observed national P-Valufe. 
The leA-h&nd ^ides of^ the* balancing equations involve weighted sapple , ^ 
estiml^tes of populati'on counts in the one-way and two-way margins bf 
NAEP's Region by.STOC, by Sex, by Color, and by Parents' Education classifi 
cation* The right-'hand sides of the balancing equatior^s involve estimated 
counts of correct responses from the five one-way margins. Suppose we let 
X^jjj^ denote ^ 1 x 22 row vector for studentf^k (adul^t or oMt-of-schcroX 17) 
ip. sohool-j (segment) of PSU-i in stratum-h with the firat element equal 1 
for all respondents*-hijk and the remaining 21 elements taking values 1 or 
0 depending on the respondent's membership in the 21^ subgroups formed by 
NAEP's reportir^g categoric? (4 Regions' +^ 7 STOCs + 2 Sexes + 3 Colors -f- 
5 Parents* Education class^es). Recalling that Y is 1 if^ respondent 
(hijk) answers correctly and 0 otherwise > ve can specify the balancing 
equations prior «to substitution f^ith the restrictions as ' ^ ^ 



(X^X)S -< (X^Y) * (2.4) 



wfiere 



and 




1^^. <P, iR(l)^ ... iR(4), At(l) 4Xt7),^AS(l) AS (2), 
AC(1). iC(3), iS(l) •:..dS(5) > / 



As, we. have noted in^chapter 1, the balancing equations In equation 2*3, 
are not linearly Independent since/ the sum of the 2nd through trhe 5tb 
equations equals the 1st as do the 6th through the 12 th^ the 13l:h and 14th, 
15th through l^th, and the 18th through 22nd. To provide for a unj^que 
so].utlon and a^ the^same time force^^e Intercept P to eqilal the observed 
natlonarl P-*value the final equation In each of the five blocks, which 
correspond ;o the :^ve reporting variables, are replaced bV a linear 
restriction^ on that var^able^s balanced effects. For exampYe, the fifth 
equation In equation 2J^ is replaced by 

ft(14+H-)AR(l) + M(2-f^)AR(2) + M(3-Hi-f)AR(^) 

+ M(4+4-H-)AR(4)'- ()• ^ (2*5) 

This substitution can be accomplished by replacing the fifth row In ' . 

T . ' ■ ' 

^^Ijfe^rjfe^ with a (1 X 22) row vector with all elements except the second 

through the fifth $et to sero. The^our elemei^ts 1^ columns two through 

/ 

five qf the new fifth row take the values one or zero to Indicate membership 
In regions 1. through 4 successively. The fifth* row of TX j^j^jj^ \±jk^ 
set to zero for every resf o^dent-(hljk) . When properly v^elghted and 
^tnmed. It Is dear that ^he new flft^ row of our individual balancing 
eq.uatlons will yield the restriction eqOcttlon 2.5. Similar substitutions 
of ro*s 12,-14, 17, ai)d 22 with the linear equations In equation 1.20 

produces NAEP*s restricted set of balancing equations. In 6ur further. 

T T ' ' ' 

treatment of balancing, (X X), ... and (X It) - ... will rdpresexii the restricted 

respond(ei^-(hljK) contributions to the left-^and right-hand , sides of the 

balancing equations. Substituting these Independent llpear restrictions 

for the redundent^ rows of (X X) and setting the corresponding rows of^(X Y) 

to zero allows one to specify the balancedl fit uniquely as 

-6 « (X^X)'-^ (X'^^Y) , ' , - . . (2 6; ■ 



wnere 



' V H ^ \y - ' _ 

. (X^X),,^„ - Z t t If W (X^X) . 
"=^22 h-1 1-1>^ k-1 'K^^** ^^^^ 



- 1 T. T. T. w. ry^vv 



2^-^^ h.-l 1-1 J.l k-l ^^J'-^ '^^^'-^ ' 



la nce Estloators . * • > 

Variance EstlmEttors for P*VaIues and AP-Values 

To suppo« the prfesentatlon^o^ P-Values and AP-ValUes, measures of the 

sazibpllng yarlablllty of these stat£stlc$ vere needed. A Jackknife replica- 

tl'oits^xpcedure for estimating the sampling variance of nonlinear statistics 

from' complex multistage^ samples was tailored^ to' our design. This technique 

is easily applied to highly stratified designs with only two primaty units 

(PSUs) selected with replacement or without replacement from strata where 

the fpc (n, /N, ) can be ignored (refs, l,i2]. The Tfear 01 primary sample • 
^n n « » 

fits ^this description^ except for a few strata containing single primary 
units. These isingleton PSUs are accounted for in the following section\ 
Xo demonstrate the computational aspects of th^s technique, we can 
consider estimating the variance of a national P**Value. -First we define 
cxpanded-up PSU totals ' ' < j • 



j-1 k-l 

and ' , * 

^ hi hlj . • ^ 

; M. , - E Z. 1W > - . (2.8) 

^oialllr^g equation. 2. 2, we see that the total H^^ represents the PSU-hi 
contribution to our sample estimate o| the number Q,f l7-yedi:-ol-ds in 
stratum-h ,while Tf, . is the PSU-hi contribution to the estinated numl^er of 
l7-year-o,ld's in stj;atym-h who could answer the question .ctfrrectly. In 

terns of these expanded FSU totals, the P-Value becpmes 

-' ■ ■ - . ' . ^ 

h«l 1*1 / h"l i*l 



The Jackknlfe estltoate of f».8ay F^^, and its variance, estimator are special 
applications of the following general r^ult for a sample of H strata with ^ 
XL primary selection^ per strata [ref • 3] (with replacement ox vltttout £rom^ 
Strata such that x^/Nj^ is negligible) • . ^ 

Lec QR depict a statistic ba^ed on data from all n^ FSUs In j&ach 
stratum. Define the replication estimate 6^^^ constructed from all, the 
FSUs excluding F5U-1 In stratum-h. * These replication estimates should be 
produced as this censored FSU.had not responded; that Is, reasonable 
nonrespouse. adjustments should be used In estimating 0 vlt;}iout^'F5U (hi). 

A 

The jackknlfe psaudo-values 0, . are then formed where 
^ hi ' 

' * A * 

The j^ckknlfed alternative for 9° is , ' . 

I I ' 

h-1 i-1 



^ h-1 i-1 ^ / 



A consistent estimate of fthe variance of 0^^ Is 



>rhere 



n*'l 



e. - E e../n. 

h. hi h ' • V, 



and 



' m 

Commenting on an earlier draft of thl9 report, Dr. Davfd Ri Brllllnger 
[ref. AJ has pointed out that a psfeudo-val^ue. of the form ' \ 

> 

would be more appropriate for a stratified sample [ref. 43*^ This result 
was obtained by approximating the expectations of 9 , and Q_j^^ with 

Taylor series of the foml^ ' . , 

- ■ 25 • - 



4 



I 



- anc . 



«^ - ;i ' " - - - 

^ '2(5"') \ I I a./iu 3econd-orcer terms . • * ' 

^ 1 ' ^ • ' • * ■ • 

-'^^ 3-+ «, /(n -1) : a,/n. T second ord^r 



•** ^corid order. 



/ 




Using the series approxinations above, one riptes that 

V"/ -^^-h." *' V'^'h second, order. ' • 

Therefore, .for ^rillloger's alternative jackknifa estiaator * 

if ■ , . " • . - - ' - 

, . • F - - ' 

■ ^ h-1 ^ . 

it is clear that' . * 

Jt(S*,J,» 3 + second-order terms. 

* w ^ U*\ ^ ^^^^^ 

Applying a similar ergxinent, one can demonstrate tha^t the jack*-inife 
estimator ^^rbposed in-^equation- (2.11)' contains first-ore^? bias 
terms;* namely, ' ' 

E.(2-.J A, 3 ^ (i-p) : a, /n, + second order^, 

J - , 

The variance approximation proposed for 3*^ by Brillingejrjrras the form 

H ' » * 

^Tne variance expression above is equivalent ,to the estimator in ecuation 

' ' •<2.i:). ■ • ■ ■ ■• 

Applying Brilllnger's result to produce a jackknifed as:i;2ate qz . \ 

P - 2^, we first' coneider the case wnere all a * ^. If this vete whe 

I ^ * ' ^ n . . . 

«oase then ' , « ^ 

■ ,,, ^ ■ * .^^w) / (\i r V •. . 



ERIC 



The teplicate F^A^alues be cone 



:he t 



'-hi 



*^-h2 



fk ^ A ^ A 




The first P-Value is fotned by discarding PSU 1 in stratua-h and re- 

A 

placing its contribution to the numerator and denominator o£ P with che 
data from^its companion PSU.(h2)* The second P-Valufe is formed by dis- ^ 
carding PSU 2 In^stratum-h and replacing its contribution with that 
irom^ PSU (hi) . The jackknif e pseudo-values become 



(2.157 ' 



azi Che jackknif e F-Value is 



h»l i-1 



? Pjji/2H.- (H+1)P - K?^ 



(2.16X 



Equation 2«16 snovs that the jackknife P -Value .^s (^1) times 'the standard 



combined ratio: estimate minus H tines the simple average of the replicate 



P-Values. The variance- estl^vate for P^t^ • • 'i& 



V 



H 2 . A * ^ ^ ^ 
i-1 i»* 



(2.17a) 



Considering (P^.^-- ?^ )^/2 ai^fT^calling chat ^ 




i-1 



1)P^>^^^^ we nee<i not botfier with the pseudo-values; chat is, 



"h* 

2 



(2.18a) 



27 



24 



t 



For n. - 2 a convenient simplification for the e^cpresslon In equation 

\\ * • * • 

2:18a is • ' . . - . 

*The dicipl^led fom for the Jackknlfe varlamre estizsator In equation 
2.17a becocaed 

■ • " H ' 

, An a^logous application of this technique produces AP-Values from 

replicates fomed by successively deleting PSUs and replacing their con- 

* * . * 

trlbutlons with data from J^helr companton FSU. If these replicate AP- 

Values^are denoted by ^^yit ^^^^ jackknlfe AP-Value is AP^j^ where 

^ * • ». * 

H 2 

' ' AP,^ - - H H j: iP . ,/2H 

h-1 i-1 "^^ 



I: 



with variance estimator 

I • 

; for those unfamiliar with the j^ickknlfe linearization technique 
deitribed above, it may be of interest to note the relationship between 
^rar«.(PTv) in-fqUation 2.17 and the standard Taylor series variance 
approximation ?or a combined ratio £re%i 53 • If ve i^t - ^^hl * ^2^ 
and 6Mj^ " (Mj^^j^ - ^^^2^ * the^Taylor series variance approximation f6r 
. var {P} is • , • 



H 



•t 



V • '.25 
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with 



varjg(P) - r 6 S. , <2.20b) 

h"l 



and 



Exacliaing the fom of the jackknlfe replicate P-Values"ln equation 2»14, 
It Is not cicult to see that 

• A JC^ J<\ 

which leads to ^ * < 

H ' 
h»l 

. , . h«l ^ . ^ 

Ccsiparlson of the Xaylor series. and Jackknife variance estimators in 
equations 2*20 and 2.22 points ou^ the close analytic rel^atlonship be- 
tveen^'the tvo» The quantity SsLtA in the dencdnator or the jackknife 
variance expression in equation 2.22 is^he stratus**h^ contribution to 
the estimated relative variance of uhere 



rel-var {M> « h-l'^^'^i^^^ ' * " 
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• Since rei»^^vaV {M} is positive ^nd, generally d'uch saaller than 1, we can 
expect the jackknif^ variance estimator to be slightly' larger than the 

^ correspond^g Taylor series variance approximation^. One would also * 
exi^ect the^diff erince between tlie two estioators^ to dlziinish as the nun- 
ber of strata increase since each stratum's contribution would represent 
a smaller fraction of rel-^r (M) . ^ Sone-numerical comparisons of ^ the 
Taylor Series a»d jac^c^cnirfi^ variance approximations will be presented in 

. chapter '4. — : ^ ' ^ « * 

Variance Estimators ^for 'Balanced Effects 

Recialling the definitions of (X^X), ... and (X^Y)^^^, , the restricted 
, • ^ nij K nij K 

respondent- (hijk) contributiqjis to the Ifeft- and right-hand sides of ouV 
balancing equations, we begin by forming the'.expandea PSU-hl totals ^ 



,^^'^^hi " ^^l^ J- ^^^r. (It^X)^ \ (2.24a) 



afid 



These definitions^ allow us to specify the ^'uantities O^li) and (X'^'Y) as 
stratum suns of th^jE^na ' ' 

H* * 

^ h-1 " 



and 



* , H 

h*l o ✓ 



The jackktiife. replicat6; estimator for the vector of balanced effeat^s, 
which is obtained by deleting the contribution from ySL-fel a^nd replacing 



3 



It with the contribution from,PSU-h2 can be specified as the unique solu- 
tion to'^the following set of normal equations 

— « 

, , (<X^X) - 6<X^»)j^] h^j^'' I'uh) ' 6(.xh)^]^ ' (2.26) 

• where ; ^ ^ 



and 



Deleting the contribution from PSU-h2 and replacing it with the PSU-hl 
contribution results in the set of "normal equations ^ 

- * y , * 

, [(X^X) + 6<X^X)j^I - t<X^Y) + 6<X^Y)j^J ; (2;27) 



which can be solved ^or the repltiate estimate ^^^2* p ^acWc^^i^^ Ps 
values 



7 . 



\^ - (H+1)S - H6 , ' • (2.28) 

are then formed from the replj;cat^ estimators where 6 represents the^ 
estimated vector of balanced effects based on data from all PSUs< The 
^jackknif^d estlm^tgr' for 3 Is ^then ^ ^ \ ' « ' 

h ■ ' * ' ^ ^ 

* ' 0^ '- E (6^, + a J/2H " <2.29) 



(H+l)6'- HB 



To estimate the variance-cQvariance mat,rix of the jackknifed ^ , 
.balanced effects ^ we use 



^^to)c of 



varj^{6j^} : ■ 6S, ^'8,/^8^ • (2,30). 



,1 



where. ^S^ - [l^ - l^^] - ^(i^j!^ - ^..^l • ' . '^.^ ' ■ . 

Notice that oS. is a (22 x 1) coluan vector and 6 3/, the transpose of 
A I* n 

^66-^, is a (1 X 22) row vector. The resulting quantity in equation 2,30 
£s the/efore a (22 x 22) matrix of esticiated variances and covariances 
monz the 2JL balanced effects and the corresponding national JP-Value, P, 
^ In appendix the corresponding Taylor series appro^iMtion for the 
variaV^-covariance^inatrix of g is derived, ^is net hod yields the 

4 estiraator 

H • • . ' 

' . * var {3} - Z 5S. 6\ ' (2.31) 

TS. h-1 ^ 

vhd^e 



Although it is not isaaediately apparent,, there is again a close relation- 
ship between the fona of the jackknif^^nd Taylor series varianee-covar- 
^ iance est^tors* Subtracting matrix' equation 2*27 froo 2*26 and rear- 
ranging ^t>enns > one finds that 

:(X^X) 6^ - •2H(6-(X^Y)^ '^Sahy^ ' (2.32)> 



where • , . ^ } 



and 



In solving th^ set of equations In equation 2.32 for 6$. t -the difference 
between our two jackknlfe paeiido values from st'rat\ia-h> yields 

'■ ^ . 

66^ - 2B0?X)"^'(«(X^)j^ - «(X*X)^ 8.^.] . (2.33) 

Using the e^cpresslon for 6 presented in equat3.pn 2.35 shows that the 
jacicknife variance-covariance, estimator in equation 2.30 differs from 
the corresponding Taylor series estimator only to the extent that 0 . » 
£he^ average of our two replicate estimators fr^m stratum-ht differs from 
the estixnate 8 based on all the data« , As the, number of strata increases » 
phe would expect £he difference betw€4n B and 8 . to get small. For 
Kationaj^^sessaent^s Tear 01 sample design with' 104 pric^ry strata> 
there should be little difference between the two methods. 



CoQputational Considerations 

The major complication that arose ±n)applyipg the procedures intro- 
duced in the previous sections to N&tlonal Assessment data was' strata with 
only^ne PSU. To allow^hes? strata to bontribute to variance, psuedo . 
strata containing two or thre^of these singletojj PSUs wer-e formed^ This 
collapsing of ^rata was done within regions and as much a^ possible 
<«.thin SOC (Size of Cocmutjlty) superstrata. State and county names for 
these PSUs were also used in the mat^ching* When two PSUs from different ' . 
strata are collapsed, some adjustment should b^ made for the fact that 
the stratum sizes (N/^^ may be quite 'different. One such adjustment is 
to replace the stratum expansion factprs (M^l) ^nd (Nj^l) the two ^ 
singles with a conbon expansion approprd^^te for a design with 'twoJpSUs^ 
selected /from the union of 5trata-h and that is, use th6 common expan- 
sion factor (N* + Hfl)/2. When applied to our jackknlfe tie thodology, this 
adjustmejit amounts to replacing^ stratum-il's contribution to P with 



and' 



\«W-«h=il- J (2.34b) 



ave "borrowed'' f:he data from stratum-^ in terns of estiisated numbecrs 
ot^l7-year«-olds per PSU (aj and estlaated nuabers of correct respondents 

(y^), but hav6 retained the nunber of pSCs appropriate for stratua-h. 
The contribution froa stratum-Z is slailarly rep^ced by 5traj\ia-h data, 
but its nunber of PSUs (N^) is retained. Tlje adjusted replicate P-Value;s 

for collapsing singleton strata-h and are therefore: 

The resulting squared diffeAnce between V^^ and P^^ divided by four is 
a conservative estlaate (ovete^tlaate) of the variance contribution fi^oa 
strata h ^nd -il. When an odd .nuaber of singleton PSUs was available within 
a major region,^ by SOC s.tr^tua» the last three singletons were used to fora 
three pseudo strata, each comprising one of thS possible pairings aaong the 
three units*. The var4.ance contribution froa three ^sin^etons was estii^ted 
by adding the three squared differences divided by eight. The division by 
*eight results froa^^he fact that each of the three PSUs is accounted for 
in two pf the squared differences* , ^ * 

» An alternative str^tua size adjustment, which requires no knowledge 
of the separate stratum sizes, N. , uses the estimated student population 
from tl)^ singleton »»^ata in the adjustment* Assuming that the PSUs in 
the two collapsed Strata all contain approximately the same nuaber of students, 
say M, then the sua of weights for a singleton strata'^h estimates 

^ , h . « y . ^ 

I M, , « N, H - N, M ' (2.36) 
_ \ • n n h 



When the h*ch stratum is discarded in^the replicate P-Value estimation, its 

contribution to the nunerator is replaced by its estimated population siaes 
-A _ . 

(M^) tines £he esticated proportion correct £roa stratu3t-Jli that is ' 



♦ % 
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: . ' ^ , > ' ' 

No change Is taade Co, che denomliiacor ^ch chl^ adjuscaenc. Suc^ jin adjust- 
ment fdrceg. an equality of PSU slzes^L which is not achieved for two legitimate 
selections from the same ^^tratufiii and- therefore would seem^o upderestimale 
the variance in tl^ respect. As long as collapsing is not extensive) the 
differential effect of the ,two alternatives i,s probably negligible. 

Aside from the problems surrounding stratum collapsing^ the application 
df the Jackknife technique to National Assessnent data was straight^forward. 
In. one pass through the data tape, ^^ums of wights for correct respof^ses and 
total sus:5 of weights were computed within each PSU for a specified cross- 
classification of subpopulations. Consider or example, the cross-classifi- 
cation of Region, STOC, Sex, Color, ''and Parents* Education,'' yielding a^five 
dimendion^ table (4x7x2x3x4), each cell of which get;^ a sum of 
weights correct (r, s, t, u, v) and a total sum of weights (rstuv) 
for each PSU (hi). JaU of the P-Values> AE-Values, and their variance 
estimators can be easily computed from sums and differences of these stored 
quantities. ^ The balanced effects are functions of /one- and two-^ay marginal 
Sums from the M and Y tables. The variances and crovariances of the 8s are , 
formed from within- stra turn ^SU differences aaong^ these one- and two-way 
marginal totals. 

.^ile the jackknife replication procedure was first introducied by 
Qupouille [ref. 1], as a technique for reducing sampling bias in nonlinear 
8tatl!stics, this feature is probably not of primary Importance in large ^ 
samples such as National Assessment, since combined-ratio type estisiators 
like HAEP*8 P-Values, AP-Values, and Balance^ Effects should be relativrely 
free^of sampling^ bia^^.^jSome empirical JustificatioSL for this oontention 
can be gained by cont;^8tlng the jackknifed versions of these statistics . 
with the original estimates. For most of NA£P|s reported .statistics, these 
differenced^'sft^ negligible. For t^hese reasons. National /Assessment reports 
unjackknifed statistics along vltb the assogiated Jackknife variance 
estimator. ' , ' ^ ^ ^ 

In the following section a summary analysis of Yeair Ol sampling errors 
is] presented. These results are extracted from a paper by Chroo^, Moore, 



anjl Clecaer trefc'6I, sThe results are presented in ems of design effects 
or the natio of JJAEP Variance 'e^iiaatjfi? to sirsple Mndoa sanpling variances. 

SumaYy Analysis of Year 01 SaOTling Errors ' *6 ' * 

^^ilatiOnal design effects were estinated %y Chronr/ er a!, for 149 science 
an(^ writing/ P-values. The ajBdian design effect estimate for tne 1-^9 exercises . 
examined was 2.38, with the majority falling between 1.50 and Table 
2-1 shows that 82 perc-ent were 3550 or less, and 94 percent were 4.0^, or 
less. Table 2-2 presents median national design feffects and ranges in 
national design effects for various subgroups of exercises classified bv 
age groupf administration mode, and subject matter ^rea. 

Design effects for groflp-administe'red exercises were highj^ than ^^o^e 
for ifidividuaLly administered exercises due to more cluste]fing of the sample ^ 
respondents. Each group package was administered oQce in each PSU to a 
iroup of 12 students selected from a single school, For individual packages, 
the 9 respondents. selected from each PSU were spread across several schools. 

The estimated design effects for 13-year-olds were smaller than those 
for 9-year-olds, while the 17-year-old exercise effects were smaller than 
Jt^those for either 9-^ Dr l^year-olds. A plausibl^j^xplanation for such a 
. trend is that high schools are more het€^rggeneoiisin terms of ^udents than 
are junior high schools, and junior high schools. are more heterogeneous than 
the elementary schoola. * " y ♦ 

Mediant design effects for size of community (SQC) . subpopulations 
I .defined by poststratizication are shown in table 2-3. As with national 
^sign effects, the median effects for SOC subpopul^.tions .are higher for 
g»7up-administered exercises thao ,fQr individually administered exercises. 
There is possibly a tendency for metropolitan and urban area median design ^ . 
effects to be smaller than those for more sparsely populated medium city 
and rural (small place) subpopulations. ^ 

The, design effects discussed in "the (Shromy pap?r reflect the combined 

effecxs of clustering qf tite sample, xinequai weighting ol,^aapie t-Q&p^nd^v^Ce^ 

stratification, and other sample design and estfiiiatiofi factors. The effe*ct 
of unequal weighting of sample respondents was estj^teo to be from 1,3 to 
1.6, depending upon <he exercise. * ^ 
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Table 2-1. Distribution of natiorial* design effects 



Design Effect Huaber Per^ 



< 1.00 , 1 ^ , u, 

1.00 1.50 'l6 IIX 

1,51 - 2.00 ' 29 m 

2.01 - 2.50 J..: 43 . 30% 

2.51 - 3.00 ' ^- -. 32 21% 

3.01 - 3.50 ' ^8 ^ 52 

3.51 - 4.00 So 7Z 



4.01 - 4.50 ^ 5 . " 3Z 

4.51 -::5.00 - 5 2Z 

> - 5.00 2 IZ 

Total 149 100% 



Table 2-2. Median design effects for natd»onal P-^alue estloatea. 















Administration 


Subject 


Kumber of 


Median 


Range of 


Mean 


' Mode- 


Area 


Exercides 


Design * 


Design 


Kuober of_ 








Effects 


Effects 


Respondents 



Age 



9 
13 
17 


Group 
Group 
Group 


Science 
Science 
Science 


30 
27 
10 


■ 2.68 
' 2.26 
' 1.81 


*1.92-4.94 
1.^31-6.01 
, .90-2.51. 


2,442- 

2,415 

2,122, 


17 

26 to 


Individual 
35 < . 
Individual 


^Science 
Science 


1. 

1 

16^ 


1.13 
2.57 


• * 

1.38-4.08 • 


579 
878 


9 

13 „ 


Group * 
Group ^, 


Writing 
, Writing 


24 
5 


2.81 
4.36 


1.51-3.80 
\ 1.93-10.88 


2,426 V 
_2,416 _ 


9 

13 


Individual 
Individual 


Writing 
Writing . 


'23 

A 


2.21 
' 1.89. 


1.45-2.68 
^ 1.24-2.88 


1,817 
1,863 
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Table 2-3* 'Median design effects for sire of comunity (SOC) 
subpopulation ?-value estinates ^ 









* 




Median Design Effects 


> 






« 

Nuabet of 




for SOC Categories 






AdiQinlst ration 


Subj ect 




Urban 


Meaiun 


Small 


Age ^ 


Mode / 


Area 


'Exercises 



Big City Fringe 


City 


Place 


97 


Group 


. Science 


• 

30 


2.26 


2.01 


2.56 ■ 


3.38 


'13 


Group 


Scienc e 


27 


2.43 


2.20 

•h 


2.14 


1.90 


26 to 










M 






35 


Individual 


Sentence 


16- 


1.91 


' 2.25 


1.47 


1.86 , 


9 


Group 


Writing 


t 

24 - 


2.04 


2.18 


2.41 


2.86 


13 


Group 


Writing 


5 • 

* 


3»79 


2.95 


3.82 


3.69 


9 


• 

Individual 


.Writing 


. 13 


1,75 


1.97 


2,66 


1.91 


13 


Individual 


Writing 


'23 


1,22 


1.37 . 


1.75 


2.38 



Design effects for adults 26 to 35 years of age were about equal to 
those for 9-year-olds, possibly reflecting a similar intracluster correZ^ation 
for the household sample due ^ sziall, coinpact clusters and variable housing 
patterns within PSUs. % 

No apparent difference was observed between design effects for science 
and wRting exercises ^ This comparison is tenuous because of the small^ 
' number of group-administered writing exercises and the fact that n^^divid- 
ually administered science exjercises were examined in the Chromy ^udy. 

Tables 2-3, 2-4, and 2-5 present *iaedian design effects for suT>populations 
defined by regional strata^ and for sex and si2e of coinmunity subpopulations 
defined by poststratif ication. Median design effects for subpopulation 
estimates are of about the same magnitude or slightly smaller than the 
median effects for national estimates. * 
_ y The largest median design affects for 9_r and 13-year-old waiting 
exercises seeo to occur in the Southeast regitsn (table 2-3). 

4 

No consistent trend was noted among, the median design effects for/ 
males and females (see tabl^ 2-4). 
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Table 2-4. Median ^eSign effects for regional subpopulation P-^valup estimates 

(d- and l3-year-olds only) * " * • 



• ^ Median Design Effects by Region 
Admintstra ti on Subj ec t NmiihfirM^ \ 

Exercise^ Northeast Southeast Central West 



Age 


Mode 


Area 




Group 


'Writing 


•13 


Grotjp 


Writing 


9 


Individual 


Writing 


13 


Individual 


.Writing 



13 
23 



1.89 
3.05 

2.34 
1.64 



2.93 
3.65 

1.30 
2.11 



2.32 
3.50 

1.85 
1.61 



2.65 
2.65 

2.17 
1.35 



\ 

Median design effects .for sex subpopulation P-value^ estimates 



Table 2-5. 



-Ase 


Adminis t rat ion 
Mode 


Subj ect 
Area ^ 


jfumber o£ 
Exercises 


Median Desl;^ Effects bv Sex 

# 

Hales Females 


13 


N 

Group , 


Science 


20 ' . 


2.57 


2.25 














26 to 


- 


t 








35 


Individual 


Science 


15 


2.08 


2.20 


9 


Group 


Writings 


. 24 


' 2.74 


2.54 


13 


. Group 


Writing 
• 


5 


2.95 


4.38 


<■ 


Individual 


.Writing 


13 


• ^.27- 


2.03 


13 


Individual 


Writing, 


23 


1.80 


1.84 






• 









Some major revisions in the National Assessment sample design occurred 
in Year 02, The first principal change involved dout^ing the within-PSU 
sample size and halving the number of primary units. The planned number of 
administrations per individual package was increased to 20 p^r PSU* ^ 

t 4 * 

The second major change Involved the use oi controlled selection of 
the primary sample to permit Stratification by State as well as by the 
previously discussed set of stratification yariables. The inplicatiot^s of 
thi^ second change for variance estinlation in Year 02 are explored in the 
next two chapters* ^ - 
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CONTROLLED SELSCTION: niPLICATIOp FOR YEAR 02 VARIANCES 

^Introduction to Cdatrolled Selection • 

Controlled selection can be viewed a probability proportional to 
size (?PS) , wlthou^iirtpl^c sampling schetne fbr selecting prlmry , 

sanpiing unltV ^pSUs) subject to 6ontrols beyond wha£ Is possible with \ 
stratified random sanplliic. Stratified, raiuiom samples, where the sampld" 
sizes in t^ie ^various straxa ire required to be proportional to corre- 
sponding* strata slzeA, are generally ^ote efficient, than pure^-y r^don 
samples.. The effectiveness of such stratification Is increased as the 
number of strata Increases. To take full advantage of the potential gal^* 



i 



from stratification* and to guarantee representation for various subpopu- 
lations (donains) of interest, Gooc^n and Kish [ref . 1] developed con- , 

trolled selecticfh as :a neans of allocating prinarv units to strata pro- 

/■ 

• . Portional to size vhen the nuabor of units was saaller than the nunsber 
. > • • . ' 'k , 

substrata generated. Jessen [ref. 2] in his recent paper* on "Proba- . 

\^-^y Saspltog wi^th llarginal Constraints" presents ar? algoritha for * 

' , selecting prinary units wiah Stratification- in several d^.rectiorisT~^ 

-^I^^ess and Srikantan {ref . 3] considered coaarolled selection flesigns " 

with* equal probability selection of PSUs within control cells (cells of 

. the ^o-way_ stratification array) . . In -a >Ionte Carlo saapllng experinent * , 

^.^they coirfpared variance approximations fox an estimate^d ratio using t'ae / 

EfethdSs of successive ^differences, paired differences, and balanced half 

sampled, It was found that these apprtfximatlonS substantially 6vet- 

estiiaaj:ed t^e variance foi^ three of the four statistics studied. 

'TI\e suits presented i^ the following sectfions relate to variance 

.estimation fo4: a design utilizing a controlled selection algorithn to ' > • 

construct" allocation" pat terries*' As tet one of^hese patterns is chosen, ^ . . 

the r^q^rect nunber pi f i);st-stage units is selected' from each control 

cell with PPS and without rejClaceiaent . The general population structure 

and sanple design are presented in^.the fdllowlng section.* Section 3 ""^^^ 

-develops 'the familiar Horvitz--Thosiip*son [fef. 4] type estimator • for , a ^- 

,populatioif totral^d derive^ an analytic expression for the variance 

^ of such a lineal^ statistic. Section 4 gives condltlohs on the set of 
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allocation patterns and the subsequent PSU selection scheme,' which pro^ 
vide for unbiased variance estimation. The appropriate Yates-Grundy 
(Y-G) (ref. 5] type vaiMance'esCinacor is shown to be unbiased when the 

^dforetaentioned conditions are met. Chapter 4 describes a computer simu- 
lation model used to genexJj[lte data for a *Monte Carlo sar?>ling exp^eriment 
patterned after the Research Triangle Institute's sttrvey design for Year 
02 of National Asse^^nt. Three variance approximations -are proposed 
in chapter 4 as alternatives to the (Y-G) estimator. Empirical result! 
of the Monte Carlo ^ study are presented. The bias^ me'an* square etror, 
and distributional properties of foup alternative variance estimators 

. for a ratio statistic are studied* . 

General Population Structure and the Sample Pesign. - '- - - -v- 

^ Cojisider a population of first4tage listing units, which have been 
stratified In two directions. If r - 1(1)R and c - 1(1)C denote levels 
of .thfi row and column ^C«^t^ication variables, then N^^ will represent 
-the number of listing units, in cell (rc) of this two-way stratification 
array. Let Y^^j^ be a characteristic of interest possessed by the k-th 
Using 'unit in cell (rc). Suppose that X,^^ is a size measure for list- 

^rck 'represents a variable that -is knovn for 
3M.*'^.'«'Tri)N^j. listing units Ijl cell *(rc) and is assuifed to have posi- 
tive correlation with the unknown 'variable of interest Y^^j.. The rela- 
tive si2e of cell (rc) .is, therefore, # ' 

where a "plus" replacing a subscript indicates summation over the levje^a* 
of that subscrijjt. An allocation strictly proportional to X pf n primary 
san^iing units (PSUsJ to the R^cells^rof our two-w^j^ array' would yield a 
fractional simple size n^^ - nA^^^ for "control cell"* (rc) . 

. Various' algorithms, which will be collectively referred to as con- 
trolled selection schemes, yield' samples with an expected allocation of 
H^i^tstrictly proportional .to their measures of siRe., These ' 
algorithlj^produce a set of S allocation patterns with the s-th, pa^em 
consisting of a set of integer allocations in(a)i for r - l(l)R and 

c - l(l)C)-. Each of the S patterns ^s a selection probability (a ) 



assigned to it, such that the e:cpected sasrple size for any cell over all 
patterns ^is 

; » * ' s . ' - ' > 

s"i rc 4 ^ 

* 

the strictly proportional 'allocation. A'dditional cell and aarginal con- 
straints are usually inposed upon the ^location patterns; for eKample, 
the cell idlocations n(s)^are required to satisfy the following sets of 



rc 



inequalities f 



I - n_ I < 1. (3.3a> 

rd * • 

^ » « 

. ' t I n(s> - n^^ i < 1, •(S.Sb) 

y J n(s) - n^ I < 1. - (S.lc) 

^ Thes^ inequalities require that the integer allocations 'to cells, coluina ^ 
margins, row nargins deviate from the strictly proportional alloca- 
tions by. less than one PSU. 

We will consider sacples with n(s) prinary saapling units selected 



rc 



without replacenent and with probabilities strictly proportional to size. 

' That is, if the s-th pattern is chos6n, then n(3) ?SU3 are selected fron 
' . ^' rc ' 

control cell (rc) with prob^ilities, ' ' ' 

I- ' • 

_ . » rck rc rc 

and without replacement, The unconditional probability over all patterns 
for selecting first-stage listing unit (rck) is, therefore, 

With Y^^^ denoting ^the vallate value ♦jf^interesV^ssociated wd^t^i listing 
ifnit <rck), we will be concerned with estimation for the population total 
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- EstltaatlOn Theory . , " . 

The following Horviti^Tlioinpson estimator for the population jtotal 
* (Y) will be considj«ed In^ subsequent seetions/ 



T 



• - , R C rc 

/ y- 2 2 .2 . . . {3.7a) 

r-1 c-1 k-1 ; *^ 

Kotice that the sunsnation In k is oyer those n(s) (possibly zero) listing 



units selectedr£r^' ceU (tc) . The * over variate value Y^^^ Indicates 

an estimate* base,d on subsequent stages of sanpling. Recall that * , is 

' rck 
the utfcondltional probability of selecting the listing unit (rck) as • 

. defined Inequation (3.5). " ' 

In ^art of the discii^sion that^foliows, 'it will be convenie^ to use 
a single subscript^say ^ * 1(1)1*, to.indexthc two-way array of control 
cells. ' This allows one to'wtite 

in place of equation (3. '6^ "and / . " 

- * > . ^n(s) > 

■ i-l k-1 £k 

. in 'place of equation (3. 7a)^ ■ ' '* " 

Assuming, that the withia-PSU stages of saiiq)ling' lead to unbiased estimates 
of the PSU totals, it Is easy to show that Y is unbiased/' Notice first 
that Y can be rewritten as, * . 

* L * i ^ ' " 



I n(s) Y /n , ■ (3,7c) 

£-1 i * * 



vhere 



• n(s) . . 



i .A 



is the tabiased Horyitz-Thonpson esclaator for the c 



total 



• ^ *- ^j^^ • Taking, the conditional eypectatiot/ of Y given the 
allocations n(s) from pattern (s) we find, 

* E{Y]n(s)} - I n(3) Y /n,. 
£ £-1 i * * 

^ ' S * , 

Recalling the definition of n« s L a n(s) = £ {ft(s)}, one sees that 
^ s-1 ^ Z ^ ^ 

Y is ^ndeed unbiased.- • 

To derive the variance of Y, the following partitioning will be 
usefxil: 




Var(Y}'- Var[E{Yjii,(s):-] + EtVaicYjn. (s)}] . 



(3.9) 



. L 



The first tera in equation (3.9) is 7ar{ Z n(s)Y^/ii^} from equation (3.7). 



4* 

Therafore, 



2-1 I 



Var[E{Y|n,(s)>] - I Var{n(3); YVnf + 

2-1 % % 



9 

♦ _ t 



L ■ I Cov{n(3),^(s)^ Y.Y,,/5,n,, . 
^-1 V^t I £ * * 



(^0) • 



Letting' 



E{n(3) n(s)} = l a^'n(s) n(s) S n^^, , 



^ .c • 



s-1 



L L ^ • 

and recalling that Z n(s) - Z n - n, we'fiad that 

X-1 2 Z-i * 



: Cov{n(3), n(s); : (n,' , -n,n.,) - -Var{n(3)}. (3.11>- 

- . * 
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The result In equation <3^*11) allows on^ to write the betveen-contrql-cell 
contribution to equation (3.9) in a form reminiscent of the familiar Yates- 
• Grundy variance expression; naxDely^' 



• L • Y ' Y \ 

The variance form in equation -iC?.!^ shows cl«arly that the between-cell. 
source of' variation In ,Y Is Pintlglzed vhen n^ (the esp^ted aaiq)le size ' 
for ceil strictly .proportional to Y. (the cell Jiftotal) . 

Returning to the second tern in our partitioning of Var(Y), equation 
(3.9) we see that,' 

- ' L <> - A ♦ 

Var{Y|n(8)} - Z {n(s)/n- }^ Var{Y, |n(a)} . f3.13) 
£ £-1 Jl * ^ * Z 

This result is an izsnedi^te consequence of equation (3.8)/ and the fact 
th^ PSUs from a particular control cell, (Z), are selected independently 
of those from any other cellj'ti')- If 'r(s) denotes the jctint inclusion 

probability for listing units k^and' k' from cell (Z) when n(s) PSUs are 

/• . * • 

selected, then ' 

• « L ^£ • /Y Y 

Var{V|n(s)) - Z {n(s)/nj^ Z Z ' {:r(s)ii(8)-ii(s^ } f -f^^ - 44 
I l"! • I * k-1 k'-k+l Ik IW Zkk' I ,^:f{ 

+ Z inAs)/n,r Z a;, /vis). ' (3.14) 

.1-1 * , k-1 ik 

2 ^ ' ' ■ 

where denotes the conditional variance of the estimated PSU total 

^^^^ ^^^^ listing unit (Zk) belongs to the ^flr^t-stage sample. - 

Since the conditional Inclusion probability 7r(s) « {n(s)/n.}ir^, where i 

£k I ^ , 

is the corresponding unconditional inclusion probability, one can 

recast eqjiation (3.14) as ^ 
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£ i-1 k-1 k'«k+l ik iik' Zkk' A '2M "Zk' } 



Letting 



L 

^ ikk' s-1 * ikk' ^r; 



i 

denote the unconditional Joint Inclusion pr.obablllt7 for listing units k 
and k* f^oa control cell (Z) and defining , 



SV, = Rel-Var {n(s)} - Van>(s)}/n, , 

> 



the e:cpectati^n of equation (3.15) becoaes 

: , : 



*ElVar{V:n(s)}] - Z Z Z < <^^V^) Wik'-''7kk*U - " 

• Jl-1 k-1 k'-k+l * *ic /-KK .^j^.y 

■ . • . ■ ■ •■ ' 

2-1 k-1 . 

Finally, cozablning the results in equations (3.12) and' (3.16) » we can specj.£y 

the variance of Y as follows; ^ ' ^ ^' 

• ^ - - 



. L-1 L Hi 
-Var<Y) - Z I <n,n. ,-n ,) I - — 
;.-l"i'-2+l ^* V'^Z \'/ 



2 
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1-X k-X k»-«fcfl f 
,t-l k-1 ^'^ ^'^ 



2 



,(3.17) " 



.Although the ea^resston. in equation (3. 17) neatljt.-pdiW.tions the variance 

of Y Into coi^onents due to bptween-cell, between-PSU-irtthin-feell, and 

' ' ' » 

vlthln-FSU variability, a sure co^act form, xdiich cfitiblnes the first tvo 

t^rms above, provides wre .Insight into^ow this variance night ba esti- 

liated. 'If »^ij.£tjjt denotes the.tmconditional probability that PSU (it) ^ 

of cell (2) and PSU (k') of..c«ll (1') both belong to the saaple where 

Aj'llV.then . . J • , 



£k;£'k 



tut 



Z a ir(s)ii(s) 
8-1 ® £k £*k* 



« »£ a n(s) A n(s) A , , 
s-1 ® £ 



■ "^U* \k ^£-k' • • 

• ' } 

Having defined this betveen-cell Joint Inclusion probability, one can view 
the first tvo stages of sai^llng (patterns and F^s given the pattern) as 

a without ^-replacement selection of n ^ I n^ FSUs with varying. Inclusion 

probabilities and' with Joint inclusion probabilities ^j^^j£tj^t ^ere, 
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This leads on<,^ the variance expression. 
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is 



(3.19> 



^* 1 



L "2 



' +22 «L/-.v • . * . . C3.20) 

Using :the definition in equation X3^*19^), it is. not difficult to show 
tliat ths first term in equation (3.20) expands int;o the between*cell and 
between-PSU-vithin-cell contributions of equation (3*17). In the following 
section, the faniliar Yates-Grundy variance form in j^uation ,(3. 20^) will 
be e^loited to produce an estinator fpr Var(?}. 

Variance Estimation ~' - . - * 



IF an un^iayed estimate-, say j^^, of xhe r/tthdtn-PSU yariance'lff 
availabl^roa each sanpled PSU, then, 

n(s) ' . 



^Var (Y) 



f) -V'' ^2 2 / W£'k'""jlka'k' \/^Xk ^i'k' \ 

• 2-1 k-1 (2'k')¥(2k) \ 'ik,£'k' /V^lt "z'k' / 

n(s)- 

+ 22 c;. ' (3.21) 
a-lk-1 

is an unbiased esti&ator for Ver(Y) when ^/^u.^iui ^ ^ for alll pairs 

(Jlk; of listing units in th^e frase. This last condition requires 

* 

that each pair of listing units in the frame has a chance of being in 
the sample and is the downfall of mo^t controIled-s^lecti<^n designs when 
it comes to variance estimation. To satisfy this condition^ the set df 
allocation patterns ^ust be such that: 

1. n^«i > 0 for all Aairs- of nonempty cells {iV) in the ^ 
two-way stratification array. This implies thacy^ach pair 
^ • 6r control cfills 1^ represented in, at loast one of the 
allocation* patterns. 



Z> If (cbe nuaiber of listing unUjs in cell i) exceeds one, then 
we require chat Pr{n (s) ^ 2} > That is, if a cell contains " 

two or Bore listing tmits, then at least one pattern Bust assign 
two PSUs to .the cell* • • ^ 

Meeting the first of these two conditions inresents no major difficulty. 
Algorithms like" Jessen' s - [y af^. 3] "Hethgrf 3" geperate an enlarged set of 
allocation patterns, which .satisfy ^be inequej^ity constraints in equatifon 
(3#1) and at the same time give^all pairs of noafempty cells a positive 
probability of being In the saiq>le* The second requirement for unbiased^ 
Variance estimation*^ more severe, sinc6 it nms contrary to the basic 
advantage of controlled selection designs; that is, having more control 
cells than PSUs. If a ce^l^ expected allocation is l^s^han one, 
although it contains two or w>re listing units^^*^ 2), then the cell 
lne9u^tx_X A(sJ.-j^^^[_jLXjk^qAiatlo^ not allow n( s) to be two 

or more* One way of solving" this problem would be ^ to restrict our atten- 
tion to designs with > 1 for all dells wltH ^ i^ut this would elimi- 
nate most of the sitxiations where control beyond stratification is desired. 
A bore acceptable solution allows some patterns^ with n|s) « 2 when n^ < 1. 

Although the celi inequality Is violated for cells with ejected alloca- 
tions less tim 1, our experiencS==4ndicates that ^t shoulJ stUU be possible 
to satisfy the marglixal cor^str^cj;, as long^as the expected marginal 
allocations excee^^d two PSUs» - 

When some of the" pairs* <ik;i'lc') of lis^^ units have no chance of 
appearing In the sas^le, the estimator in equation (3.21) will underestimate 



f 



e true .variance by ax; azaounr> 




where Z denotes stssDation oVer those pairs of listing units 
(ikji'k') 

(tkji'k*) such that '^j^.j^tj^i " 0, The portion of this downward bias due 

to singleton cells (cells with >, 2 but Pr{n(s) >. 2} ■ 0) t^n be expressed 

as' . . j 
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• • 2 

* 

(s) 

wnere Z ' represents suinaation over all singleton cells. That part of 

equation (3*22) due to pairs *of. cells* having no chance o£ being in tbe sample 
is, - r 

Z n.n., E E A,. A,, , 1 

^ ^ k-i k'-i yik *t'k' 




vhere I" denotes summation over those pairs o£ control cells vith no 

chance of beii{g in the sariple. As suggested earlier, this source o£ bias 

•I' 

cnii be* elininated by adopting .an algarithia like Jesson*s "Method 3," which 
* 

guarantees that all noneiiipty pairs o£ cells have a dhance of appearing in 
the saople* ' "TN^ ^ ^ 

The underestii^ation tliat occurs as a result ^^singleton c&lls, equation 
(3*23), can be compensated for by a procedure analogous to collapsing strata 
when a single FSU Is selected per stratuzs. One such ^bollapsing schen^. 

Involving successive diffeirences, takes the following fora: 

* 

m(s) • i 

with ifl(s) + 1 = 1, " , 



wher^ t * 1(1) a(s) indexes the singleton cells fron colutm '(c) represented # * 

in pattern (s), and » ^tcl^'^^^c^^ denotes the expanded PSt total fron 

singleton cell ^tc) This schene presumes that the coliimn classification 
represents the more important stratification dimension^ Snd that ^ne row 

^- ' 
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categories cato be arranged in a Circular array with aQjacent categories 
assumed to be taore alike than nonadjacenU cat^egories* Adding this term 
to equatfoa (3.21) and assuming for the moment that n^^^, > 0 for all' pair^ of 
cells, the bias becomes, 

m(s) 



• * 



S C c /Y Y \ ^ ' 

^ S M /2 + I. .J,.' (3.26). 



The withln-PSU bias renainlAg in equation 3.26 can be eliminated by sub- 



tracting, ^ ^ 



^ n(8) 



l^^^ ^ ^L/^v . (3.27) 
!> k-1 



from 'equation (3.21) • If Z denotes summatipn over nonsingleton cells, then 
equation (3.21) becomes with the bias adjustments in eqxxations (3.25) and (3.27), 

m(s) * / . 

^iiii ^*^^tci-^^i^yt+i.cA . (3:28) 

. / ■ n(s) - 

where v^j^j£fj^f represents the Yates-Grundy "variance weight'' in eqilation (3.21) 
and y^j^ is the expanded PSU (ik) tota^. The bias-adjusted estimator in 
equation (3.2§) overestimates the true variance of Y by the first term of 
equation (3.26). If som^jjj^irs of cells have no chance of being. In the sample, 
then the quantity J^n e^xiation (3.24) makes a nfegativcf contribution to the bias 
of equation (3*28). 



50 



52 



Finally, if the-within ?SU,variatJLon is not estiaable, as wheg^.oniy 
one second stage unit £s selected per PSU, an additional negative bias is 
incurred; namely 



2^'^ '^^ . ^ (3.29) 
X' k«l . " . 
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Chapter 4: SIMULATION STUDY OF ALTERKATIVE YEAR 02 
\ ' ' . VARIANCE- ESTLWORS , , 

. ' , ' • ■ 

Slmulatlop. Model ^ / 

In the following section, Che design of a Monte Carlo samp£ing experi- , 
. • ,, 

nent is presented. Modeled after the Research Triangle Institute's (RTI^t^) 

sample design for Year 02 of ^^ai^lonal Assessment, 1,000 samples of 31 PSUs 

(counties or groups of counties) were selected from the'^twd-way state by - , ' 

"major strata" grid shown in tgfede 4-1. Table 4-1 shows the expected PSU 

allocations (n ) and numbers'^ first-stage units in the frame (U ) for 
, rc rc 

the 15 States comprising KAEP^s western region. The seven ciajor strata 
represent a combination of size of conmUnj.ty and socioeconomic status 
categories. ' ' ' - 

In order to distribute the ^jample )^roportlonally across the major 
strata and at the same time guarantee that each State would be represented 



by at least one PSU (a NAEP requirement), controlled selectiorr was ttsed 

to generate 33 p'su allocation patterns that met these requirements. Ihe ^ ^ 

33 pattern probabilities for our design were converted into integer alio- j 

cations by multiplying each by l,.0O0. In this fashion, the number of tines ] 

a pattern was represented in the 1,000 samples was made strictly proportional 

to Jts selection probability. For each"" of the 221 f irst-stfttge listing lunits ^ > ^ 

in the ttest, an estii^te of its populati©^ of 17-year-olGS was produced 

using 1960 census projections. These estimates were used as size measures ^ 

in connection with a PP5 without replacemei^t scheme to select PSUs from the 

cells of table 4-1. ^ X 

A data vector consisting of the actual numbet of 17-year-olds and the 
'nun^er responding, correctly ^o several SAEP test exercises was generated 
for each of our 221 first-stage listing uniij^. Tnis data set was based^ on 
1970 census figures for#^umbers of 17-year-olds ana on estimated P-values 
(proportions co^rrect) by State and major stratum margins f6r the selected 
NAEP exercises. To*'produte P-values (P > consistent with#actual census 

r ' . * ' ' * 

totals and the observed- State and SOC marginal P-values ior jselected ,;iAEP ' 

year 02 exercises, an iterative proportional flitting technique was employed ^ r 

[ref. 1]. These fitted cell*P-values were used along wlth^tne listing unit / 
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4-1^ primary ^ampltr^l location sujnmary for •the West 



State' . ■ ' \ 

1. 



Stratum 



4 • 



r - 



\ * Aril. . f . -.;i6 K^) *; 

" (killf. * . 4.000 (4) a'.lSy (5) .567 (2) 

coiL^ '# .203 ay^ -wi C4) 



^096 (f) 



. Hawaii 
Tdalio 
llont* 



.868 (1) 



Hev* 



* oicla. 

ft 

\ _ Tpxns 

' * Uta|> V 

♦ ■ ♦ I 

• . Wasfu * 
. -Wyo. ''^^ "\ 
. . Totals 4^^000 (4) 

^ ••• • r 

er|c • 



.268 (1) ■ , ..•22 

^^.735.. (2)^ . , - .48 

.343 (2) •* \\!li){X). 
^.400 (6) (3) \1.10 

.447 (1) /.i*!Kl) 
*<831 (2) .366 (: 



. 0 



stratum Huaber 
5 

•V. 






Total 

PSll*8 

Allocated 
1 .(Ho.) 


' Total 

rSU 8 

In Fromc 


.372 (1) 




,628 (-2) 


1 


(3) ■ 




.150 (1) 

f 


.134 (4) 


1 




2.601 (16) 




.553 (13) 


10 


(41) * 


.099 (2) 




>.l3l'{5) 


1 


(13)- 


.132 (1) ' 




1 . 


(2) 


.404 (3r) ^ 


* a 


.596 (6) 


1 ' 


(9) 


•' .482 (4) 




.518 C5)^ 
.194 (1»^ 


^ 1 - 


(9) 


. .806 (2) 


0 


1 • 


(3) 


..144 (2) 


.148 (1) * 


.212 (4 > 


* 

1 


(10) * 


.070 XD 
■ ■*i85 (2) 


'■ .461 (4) 


. .254 (8)*" 
^ %350 (9) 


2 
1 


(14) 


.530 t8) 


1.241 (14) 


.398 (20) 


6 


(64) ' 


..' .215- (2) 


•* 


.189 (3) * 


• 1 


(7). 


!* .392 (4) . 




.4il (8) 


2 


7(16) 


..568 m 




.432 (2) 


,1 , • 


(4) • , 
(221) 


7.. 000 (50) 


2! 000 (&) 


5.000 (96) 


31 . 


















• 


\ ^ 


s 


m 


■ * 

• 


t 


V 


* 


** 

t 

4 




♦ -»-"- -■»»■ 
• • 



totals xo product 'a variate value Y , corresponding tcf the nuiaber of 
/ 4^ rcK 

/ * correc-c responses from listing unit (rck) . 

The followitig nodel was used to gcneraci a new value for i'^^j^ each - 
time listing^^tvit (rck) appeared in one of our .replicated samples ^ 



Y ,M ,+e^, 

rck rck rck rck 



(A.l) 



where the ertors e^^j^ for different listing units (k) from control cell 
(rc) are uncorrelated with zero expectation and variance 



^ rck" rck' ^ rc rck 



(4.2) 



Modei.s similar to Equation (A.l) have been used by J. Durbin [ref. 2], 
, J. N. K. Rao [ref . 3]-, and nuzaerous others to study th£ propjerties of ratio 



e^tiisators for ? 



We will restrict our attention to models with g»l anc 
rc . 2 ^ 

proceed to motivat;e a particular choice for the value of a . Our method 

' 2 ' ' . rc 

of choosing a value for c^^ will be tQ propose a plausable model for the 



sampling variance of the estimated cell F^alue P and then find a value 
of consistent with such a tcodel. 

- Suppose for the moment that Xistiiig units were selected with replace* 
ment and with probabilities sprictly proportional to known sizes M^^^^* Then 
the unbiased estima 



iror 



with 



rc 



n' (s) 

rc- 

Z ■ ] 
k-1 . 



' rck / rc 



(5) 



^rck ^rck / ^^rck 



/■ 



• (4.3) 



has variance 



Var (P ) 
s rc 



A. 



rc / rc 



(s) 
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^ * 



c 



where . . 

N 

'^■^< • * 2/ 



k-1 

I^>^ In addition, we define 



"re- ■ • 



as the wlthln-lls ting unit variance component) then, tt Is e^sy to see that 
Now, we define 



or 



rc rc / rc • tc . 

as the wlthln-llstlng unit correlatilon coefficient* These deflnltlcjns 
allow us to write * ^ • " 



s . rc - rc IP / ' ^ 



r 



With the variance formula In 4.& representing a plausable modej]L ^or 

the between-listing unit vai:latloxt In P , the. estimated proportion correct 
^ ^ * <t ^ rc ^ j£ 

from cell (rc) , our goal Is to specify a valu^ ^or a In the d^ta genera- 
^tlon model equation (4^*1) such that the average sampling variance of the j 
•ratio estimator. ^ ^ ' " . ■ 

» « • • • 
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p 



rc 



and 



rc+ 



4A.9) 

V 



rc+ 





is approxinately equal tx> equation (4 .,8). ?*ec5illing the fona of our oodel for 



Y , we note ,that 



rc^- rc rc+ ^ rc+ 



Since f (^^ck^^'rck^ " clear that 



(4* 10) 



and 



£ Var (Y O * Var^ (M + £ Var^(e 

« ^ rc-r rc s rc+ , ^ s rc+ * 



(4: 11a) 
(4. lib) 



s 



Using these two results and the Taylor series approximation toj the variance 
of a ratio;- ntoely, , . * . 



fc Var (P tf^_^ 
s rc , rc+ 



we find that 



(4.12) 



(4.13) 



Mow we can evaluate the expre8si0jBk in equation (4.13) ip terns of the data 

• * * * 

generation laodel equation (A.l) and compare the results to our variance 

qodel in equation (4.^). First we recall the Yateg-Grundy version of 



Var^ ^Sc+^' ^' 



. N ^ 



'rck 



^rck' 



(s) 



2. 



(4.14) 



^ Using thti independence of e^^j^ and £^.^1^^ altfng with the result that 



:er!c 



we have 



M 



e Var 



Recalling our specification of the error toodel In eqiiation (4.1), ve have 
assumed that with g«l " * . . 



(4.16) 



1 



^ <4kl"rW ^rc "rck ^ ^ 



(4.17) 



With thfe additional assumption that the size measures' A^^j^ entering into 
the inclusion probabilities, .^^.^(s) Wughly proportional to the actual 
listing unit totals M^^j^ { or H^^^ ^ 0^*^ KtM ^^^^^ ^rc ^^P^^^ents the ratio 
I ^rc-t^' average variance expressiqn in equation (4.16) reduces to 

e Var^ i (s) / N 1 N -M / n (s) , (4.18) 

8 rc+ * rc / / rc' rc rc+ rc / rc \ * 
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?roi3 equation 4,13 we have 



(4.19) 



Recognizing the tern in parentheses ^bove as the finite population- ^ 
correction term from a simple randofn sample and recalling our variance 
models in^ equations (4.4) aijd ^4,8), we are lead to 

- M » M P ^ (1-P ) 6 <4.20) 

^ rc rc, rc rc. rc ^ rc' rc 



where M * H ^ , / . Ti 
rc4 rc+ / , tc 



This value_of C yields the following approximate 
^ rc 



e^cpresslpn for fthe average variance of P : - 



with f (s) denoting the sampling fir action for cell (rc) in pattern {s\. 
rc ' ^ ' L ^ ^ 

A variance components analysi^ of NAEP Year 01 in-school data suggests 

an average value for the wit)\in-lisV^g ^nit correlation coefficient d^^ 

of around .015, Substituting this value for o into our expression for* 

2 J - rc 

O , we have arrived at the followiiik computer simulation model for the 

number of correct responses Y^^^^rom listing" unit (rck) : 



where | is a standard normal error and 



V 



^rck ■ ^rc »$ck " V^c. frc ^^'^r^ ^ '^^l^ck' ( 



(4.23) 
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Ta further assure^ that Y^^^^ is an acceptable estimate of the nuober of 
correct ^sponses from listing unit rck, we have required that 
^^^1^^ (^f "rck^* was ^accomplished by censoring values of C, which 

caxxse Y^^^^ to exceed these limits* In order to preserve the symmetry of 
our error distribution,, which assures that e (^^^j^Im^j^i^) " M^^j^, we 
have censored valxies of 5 in. a syianetric fashion. The. rules fo'r rejecting i 
5 are, therefore: " * . 

' 1. For P^^ > .5: 

Reject 5 if UIXl-P^^) ' . (4.23a) 

2. For P < .5: . ^ 

rc " 

Reject Ci£U|>P^^«^^/d^^. - .M4.23b) 

Only for values of P^^ dose to 0 or'l will, any censoring be required. ^ 

If, for example, we make the simplifying assumption that M , ■ M for 

V rcK rc. 

all k, then with P - .95 or .05 the limits are approximately +1,87. 

rc ^ <-* 

Values of 5 would be expected to exceed these limits about 6 percent of the 
time. ' For P-values P « ,90 or, .10 the limits become +2.72; these limits 
would be exceeded about .7 percent of the time. - ^ 

^ The model for Y^^j^ described above^deviates in twc| essential respects 
from the model used in the first version of these results presented at the 
ASA meetings in Montreal in Angust J^72. The first and probably most critical 
difference was in the specification of^th6 error variance ^(^^^1^1^^^,!^) • In 
the Montreal model, we let e^^j^ • rf^^^ % ^ith 5 a standard normal error and 



^rck - ^rc. ^.c (^^^c> \^yS^^ il-\ck<^>3- 



The average variance of e ^ under this model^s ^ 

rc+ m 



e VrfrJ '(e ^ J • M _^ P (1-P ) , (A. 25) 

s rc+ rc+ rc rc ' v-^^/ 
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t^hlcir lead^;^o the admittedly unappeaUng* result 



This unfortunate choice of error variance, should have seriously under- 
esticiated th^ 'between-listing, unit within control dell 'variation. The 
second change In the new siodeO. was td regenerate the value for ^^^^ each 
t^e that listing unit (cck) appeared In one of our replicated "samples* 
This change should cause the sinulatlori to approxinate Dore closely the 
expected level of between-'listing unit variation. , * * 

^ To build within-listing unit Variation into our sinulatlon, we first 
take note of the two stages of sampling within NAEP PSUs. The* typical XAZ? 
itt-sjthool PSgJjas two , schools representing a paftlcular group package with 
12 Students selected from each school. Suppose we let and ^^^.^^ 

denote estimates of the total number of 17-year-olds and the corresponding 
Ttxsbex of correct responses from listing unit (rck) based on our sample of 
schools and students. If we assume that schools are selected with replace- 
me;vt and with probabilities propODtlonal to- known numbers of 17-year-olds, 
say M^^^ for the % « 1(1) S^^,^ schools in the listing unit (rck) frame, 
then with simple random sampling of students wttnln schools (Ig^iorlng the 
laft stage finite population correction) ,'^we have ' 

Var (P ''-Y ,) ^sA'^'^-l) (A. 27) 

\ s xck rck/ rck (rck)Jl/ (rckija / ' 



where 



'(rck);i "rckZ ^^rck;i"?w:k' 



^rck 



represents the betwe^-school, wlthin-llstlng unit variance coaponent. The 
between-students, wlthin-school coDponent takes the fona 

S 



<, rcfc 



2jrc.U)m-,^^ ^rckZ ^ck£ W ^\ck^ • . ^'''^f 
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Mith^cSe definlrtiofiS of the within-PSU variance components, it is 
• oot difficult to feee that' 

This result allows us to define the within-school correlation coefficient 

4 I. - 

and to write 

■ 

The f^llowlug computer simulation mpdel was built in accordance with 
the variance model in equation (4 •SI): 



with 



and 



also 



\r±th 



and 



•r.. 

^ <?rc'^ i W ■ "rck.f*^ t - 1 or 2 " "(4.320 



,^ <4k'i "rck^-- "L^ck <1-Vc1c5 ^ •'^Sl ^ (4.33c). 

\ ■ - . 
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where tr}ie constant ♦OSf represents an, average value for the. quantity 
, (1 + ^^^rck^'^ "^^ ^^^^ ^rck •086^ The errors in the models above 

were agatsi obtained frosi a censored, ^^and^d norMl error generatorT 
In the Montreal siculation, we use^f errolry^ch that 



I 



rck 



(4*34a) 



and 



which leads to 



rck 



rck^ 



rtk * 



(4.34b) 




.Compared to* the ^varia^ce piode^L in equation (j5**31), this formulation woxild 
appear to-undetrepresent the within-?SU variation* * 

Variance Auoroxlaations . * - 



In addition to estixated tjotals of the numbers or 17-year*olds 
M, , , and the numbers of correat responses Y , foyr variance estimators 



for^Var(P - / M, , ,) were conputed from each sample* The first of 



these variance estimators (VI) uses the first two terms of ^tlie bias-adjusted 
Yates-Grundy type estimator in equation (3,29) with a jackkJ^fe pseudo value 



''jik 



31 P - 30 



H - n, 



■Jtk 



(4.36) 



tilling the place of 31 y^^, the corresponding pseudo value for Y^_._^ 
rne fom of the jackknife linearization, which uses 



2 P 



^ - ^y&k " yrk'> 



(4.37a) 
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and 



(4.37b)" 



mighc seem more appropriate for vatiance estimators involving squared , 
differences. Recall that this vas the form \ised wltK the Year 01 Design, ^ 
vhere there vere two FSUs selected per primary stratim. To simplify our 
calculations^ ve decided to use the linearization in eqUatio^T (4*36) for 
all four of our variance estijoators. While ve intend to present a comparison 
of alternative Jackknlfe and Taxlor«*Serie& linearizations in a subsequent 
section, at this point ve felt that the form of our estlioation equations 
vas more crucial to the cotaparison than the type, of linearization. 

The second variance estimator (V2) studied in the slmxilation vould be 

* appropriate i£ the^FSU allocation to cells vas fl^ced and FSUs vere selected 
wltfh replacement. A circul^ successive differencing scheme vas ixsed 

.^within major strata to collnse cells ^ere bnly one FSU vas selected* For 
this purpose, the* States in table- 4-1 vere/^crange4 in the following 



circiilar array 

4f ^ Alas.-HJ^h.^reg.H<;alif .-^ey.-^^jzli.-tll.M.-Krexas v 
V Hawaii^ Mont.^ Idaho^ Wvo.^ Utab*^ Colo.^ Okla. 1^ 



This particular ordering is,base4 on*geographical proximity atid represents a 

crude ^t tempt to make adjacent States in the array more alike than nonadjaceut 

States. If u*l(l) r (s) represents the ordered array of single FSU cells 

in a pattem(s) sample from column c and Z denotes siTfmnation over control 

' jl ^ * \ ' 

cells with njj(s) >^ 2, ve .have * ^ * / 

n^(s) 

; V2(s) « ^^(gj 2 (p^^ ^ )2 ^ j^^^^j ^ ^3^j2 

$f k"l J, * 

^ * 

* Z Z (P„ - Pi,!,,)^/ 2(31)^ ■ (4.38) 

C»l U"l , * 



where 



r^(s) +1 = 1 . 



— ■ • * 

Notisie that the second tens ip equation (4.38) includes all^ cells \7ith 
n^Cs) * 1 whereas tlie singleton cells in the corresponding tera o£ VlCs) 
(the Yates-Crundy estimator)* must hav^ N^j^ > 2. • 

Our third estimator (V3) ignores anj? control in the> State dlmenAon 
and computes the variance as if PSUs had been selected with replacement 
from ^e seven major strata. Suppos 
in major stratum c. Then ^ V 



f rem ^e seven major strata. Suppose\w » l(l)n (s) indexes all the tSUs 

* H c \ 
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V3(s) - 2 n (s) Z ■ (p^^ - P / In (s) ^ 1] (31)^ (4.39) 
c-1 ^ w-1 wc .c c 

/ • * . . ^ ' • ^ y^' 

The final, estimator (V4) ignores all controls » using the formula that would 

be appropriate ^f the 31 PSiJUs were an unsttatified selection with replace*- 

ment from the entire list of 221 flrst<-stage lasting units. This estimator 

is computed as follows: 

• j> . ~ — 

7 "c^^^ * -4 

n ' I ,Z (p p )^ / 31 X 30 . ' ' r- (A. 40) 
* - wc . . , 

C"fl JJ»1 , 

* , ' m \ • - 

Empirical Results " ^ • * , 

Table 4-2 ;>resents the sampling expectations^ Z?, ^nd variances, VP, 
for ^nine estimated P-values when the liontreal mode^^s applied. Ihe^ 
quantities (EY/EM) representing the ratdo of numerator and denominator 
sampling expectations show that there is ver;^ little bias due tC rat\o 
estimation of the P-values. Unf orCunately^ this is pot the ca$e with the 
four variance estimators. The bias»adjusted Yates**Grund3^ estimator tends 
to underestimate VP^ wheteas Che other three approxkiations tenJ seriously 
to* overestimate VP, The magnitude of the various estimators tends to « 
increase from VI through. V4 as one might expect;« VI makes an attempt to 
^account for the between-PSU variability properly while underestimating 
the within-PSU variability. The other Chyree estimator^, while accounting 
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-10.17 


VP " 
3.63 


, EVl 
2.97 , 


EV2 
5.43 


* > EV3 


. - EV4 
5.92 


38.09 


4.17 


2.70 • 


-6.14 




7.-63 


t 40.45 


4.24 . 


2.19 . 


'6.41 


6.'8^. 


'7.73 


62.t2 


5.34 


4.29 


■ V 
8.09 


8.75 


9. '32 


69. .14 


4.2'6 


3'. 95 


7.26' 


. .7.32 


7.95 


72*85 


3 77 * 
, 3.77 




ft' ftft 




7 OA 


77.46 


3.54\ 


3.55 


5.20 


*5.42 


6.08 


91.73 


3.05 ) 


' 2.-63 


. 2.14 


2.32 


2.64 


93.53 


3.19 


^0 • 


•2.36. 


2.56 

_ 


3.08 



Table 4-2. Bias conparisgns fgr fpur variance estloaVbrs (Montreal model) 

- p-(ey|e{) 

10.16 
38.12 
40.39 ' 

if 

62.42 
^ 69*.li\ 
_^ ^72.92 
77.52 
91.73 
93.55 



for the within-PSU variation properly, tend to ovferes^timate the between-PSU 

variation by ignoring a^y finite population correc'tions and overlooking 

• ' . - • ■» 

various' levels of control beyond stratification. 

Table 4-3 shpvs thac in cerzns of l^sc cocal error, rooc mean squared 

error, VI is superior to V2, which has a slight edge over V3. The 

tins3tatified estimator V4 performs poorly for all except the two laifgest 

P-values. The average performance of the^ four estlmap>ts Dver all nine 

P-vaXues -Is summarized in table 4-4. In terms of absolute relative bias, 

the Yates-<inmdy Jty^e estimator look3 better th^n the jbther three. When 

one looks at r ela<^ve total error, or joot mean squ^i^d error divided by 

the/advantage for vi is not as great* The-^abHity figures in tabl^ 

4-4 rBi>resent averages of ^^tlmated degrees of freedom whAe, for the, i-^th 

variance estimator, s 

df(i) = 2[EVa)]2/Var[V(±)] for i - 1, . . . ,4. (4,41) 

These stability measures relate directly to the shape of the Trlike sampling 

distributions summatized in table 4-5. *^ 

» * ^ ^ " < ' 

The avifiraged frequency dis'trlbution^ presented in table' 4-5 show the 

. proportibn of -Clmes th^t ^ * 
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Table 6-3; Root sean squared error of variance e,stinators (Montreal model) 









* EP 






10.17 

* 


3.63 " ' 


2.95 


« 

38.09 


4.17 


"3,^05 


60.65 


6,26 


2.89 


62*63 


5.36* 
* 


*3,65 


16 


6.26 


2.56 


,72.85 


3.77^ ; 


9.35 


77.66 , 




"2.66 ^ 


•'91.73 


3.05* 


1.64 




3.19J^ 




■93.53 


- 1.58 


■ * 


s 





4.70 
4.*58 
' 4.55 
6.48 
5.41 
J .38 
' 2'. 57 
H^.18 
IS^' 



* 4:^5 

- 4.69 
4.80 
-6.65 

.5.37 
5V61 
3.55 
2.14 
1.7'9 




T(i) - (P - EP) / F(i) -r(i) 
and 2-like statistics 

. ' t> . . Z - (P - EP) / /1?P, 



for "i ■ 1^2, .. .4, 



(4.42) 



(4.43) 



fall within the stated limits. Notice that the T-like statistics have ; \ 
been corrected for bias in V(ij_by applying the factor F(i) « VP/EV(i) . 
Comparing Z to the'tiomal frequjencies in the last row of table 6-5, one 
notes that th^ sjjEuaetric intervals are reasonably close, especially for 
tHe larger, more critical intervals. Some positive skewness i% observed 
in the asymmetric intervals. Of the T-like distributions, T2 appears V. 
-'closes't to Student's T with 30 d.f. Recalling the* stability measures in 

A A 

table 6-*6, one cagi see that the distributions T(l) .through T(6) become 

A a' >, 

. mor^ like Z as the stability of V(i) increases'.. As estimates of appro^>:i- 
mate "degrees of, freedom*' these stability measures afe grodts underestisiates. 
Tk^ naive appnpximation based on the number of ?5"Us seiectejfi is on th^ 
ofher hand reasonably* accurate. Before we conclude th^t V(2) is superior 
to the other 'estimators when it comes to m^ing inference about ?, It is 
important to reca^ that the results in taSle 6-5 are corrected for bias. 
If these -correctidns had not been made,*none of the estimators would yield , 
anything* res embiiiig a T distribution. 
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, Table 4-4. Average performance of variance estimators (Montreal model) 



Rel-Blaal:!;) 
Stability (df) 
Rel-Errqr (%) 



VI 
13 

3.52 
72 



V2 
50 

4.47 

101 



V3 
53 

4.64 
101 



V4 
63 

5.51 

108 



CO 



Table 4-5. Sampling distributions for normal and T-llke*8tati sties 



Proportion w/in 


±2.576 


±1.-960 


+1.645 


±1.282 ^ 


±1.036 


(-1,960,0) 


(0,1.960) 


(-1.036,0) 


(0,1.036), 


Student's T(30d£) 


.9848 


■ .9407 


.8896 


.7903 


.6915 


« 


.4703 


.3«5B ^ 


v3458 


* 

Tl 


.9367 


.8941 

« 


.8522 * 


.7662 


.6829 


■ ,4379 


.4'^62 ' 


.3323 


.3506 


T2 


* .9816 


:94"i3 


.8967 


.'8074 


.7158 


.4587 


.4827 


.3473 


.3684 


> .T3 


.^861 


"^9480 


■ .9064 


.8142 




.4623 


, .4852 


* • 

.3500 


.3711 


• T4 ■ 


. 9859; 


'.9516 


.9088 


.8204 


.7277 


.4632 


, .4883. 


.3507 


' .3770 






• .9B05 












m * 




. , -a. 


.9906 


.9246 


.8423^ 


.750^ 


.4756 


.4849 


.3702 


.380?. ^ 


Norm. Deviate . 


' -.9900 " 


.9SfiO^ 


.9000 


.8000. 


.7500 


.4750 




.3750. 


.3750 
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In, figure 4-1. ve have presented .a plot of true and estinated variances 

against their corresponding P-valuesi The curious relationship (or lack 

of any) between the true variances ana P-values is sy!aptoiiiat;ic of the 

probleias inherent in the Montreal model. We should expect a strong quadratxc 

relaciohship between ? and VP aodeled after the siziple random sampling 

case where VP. •» P XlOO ^ ?)/Tt. Tables 4-6 and 4-7 display results based 

on the new rcoliel where the withln-PSU variation in the estimated number of 

17-year-olds, say M^^^^.is set to M^^^^ (or M^^^^ with t-2). 6ne thing ve 

notice immediately about .these tables is thatT^fir^the bias-a<^sted Yate^ 

/ * 

Grundy^ estimator, Ifas b6en eliminated from consideration,^ Tne reasons we 
have excluded VI from further consideration at this time are two. The 
principal reason is' the e:ccess4.ve cost of computing VI relative to the 
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Figure 4-1. Montreal model (true variance vs. estimated variances). 
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Table 4-6. Bias comparisons for three variance 
estimators under the new model with t«2 



EP . 


* 


VP 


. 10.15 




3.21 


, 38.34 




8.29 


40.41 
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62.51 
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69.0^6 




8.08 
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77.53' 




6.22 


? 91.71 




3,73 


93.47 




3. -17 



BV2 EV3 EV4 

'S.Ol , 4.99^ . ' 5.46 

lOyS ■ 10.97 12.12 

10.84 11.18 12.01 

11.56 1?.25, . 12.81 

'10.23 10.27 ■ f 10.95 

9.38 9.15 I 9.82 

7.36 ^ 7.53 5 8.20 

2.81 2.^5 ^ \ 3.27 

2.29 ' ~ 2.51 ' 3.03 



Table 4-7. Resign effects with N-744 for new mcxiel with t-2 
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otlrer estimators. This cost is on the average 16 tiaes greater than tne 
xost of V2, the cheapest estimator. Finally^ although VI has the sizalle^t 
bia$» the^ fact that it tends to udclerestiQ^te the true variance Is dis- 
concerting* If an unbiased variance estiijiator is not available, one gener^tlly 
prefers an overesti^te^ which results in conservative inferential statements. 
Although it may be hard to judge the dollar value of a 'good? variance 
estimator^ excessive conputing cost would seem to constitute a reasonable 
excuse for eliminating one of several mediocre estimates. 

Figure A-2 plots the true variance and , the expected value of V2 (the 
least biased of the conservative estimates) against the corresponding 
P-value. The ^ew model with t * 2 exhibits the desired quadratic relatioar 
ship between VP and P. It is interesting to note that V? does vfot decli;ie 
for large values of P as fast as one might expect. This is deisonstrated 
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clearly by'^the Jump observed In design effects or DEFF v^ues for the two 
revalues (In table 4-7) exceeding 90 percent. The three variance estimators 
behave more syrametrlcally dropping below VP for F > 90 percent. The median 
differences between those design effects estimated by V2 and the true design ^ 
effects based on VP is (+) ,66^ 

Figure 4-3 shows a plot of true variance VP and expected values of V2 
(EV2) for 20 P-values, using the new model where the withiil-PSU variation_ 
in the estimated number of 17-year-olds H^^j^ is equal to M^^j^ (or ^^ck 
t"l) . This model yields a surprisingly smooth parabolic relationship 
betwe^P and VP, The slow asynaactric decline in VP for large v^ues of 
P Is again apparent with V2 dropj>ing below VP near the point p " 87.3 
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Figure 4-3. New model with fl (true variance vs. estimated variance). 
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percent. Table 4-8 shows the relationship between VP and the Expected 
values of V-2, Vr»3, and V-4 for the 20 new aodel ?-Valaes with £-1. Design 
effects D-2, D-3, and based on the three conservative variance 

estiaators, are couparsd to the true design effects (DZFT) in Xable 4-9. 
The oedian differences betveen the estimated DIFF based on V2 and tr.e true 
DEFF is, for this case, C^).64* Tables ^-10 and 4-11 show the^-^v^rage 
perfomance of our three conservative estinators for the new nodei with 
t*2, 1 respectively. Although, V4 is consistently the nost stable. of the 
three^ estinators » V2>^ends to have the smallest bias and smallest root mean 
square error as reflected in the average Rel-3ias and Kel'-Error terms. . 
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7.2p 




8.76 


8.95 


9.81 


38.18. ' 


8.40 




10.00 


10.37 


11^65 


40.32 


8.61 




10.28 


10.64 


11.47 


47.50 


8.80 




10.47 


10.83 


14.06 


55.00 


8.54 




12.40 


1'2.94 

• 


13.54 


62.43 


8.04 




.10.70 


11.43 


12.05 


69.04 


*6.89 




10.05 


10.02'- 


10.72 


72.95 

• 


6.20 




.8.75 


8.51 


9.24 


-77.51 


5.67 




e.66 ' 


6.88 


7.58 


82.00 


4.87 




6.10 


6.78 


7.51 


87.47 


3.77 




' - 3.60 


4.09 


,4.67 


91.69" ' 


3.06 




1.95 


% ■ 

'2.11 


2.43 


93.53* 


. ?.JQ 




1.52 


1.75 


2.25 


'96.00 


2.33* 


^ - _ 


.73 

t 


.79 

«% 


•i;is- 








73 


76 ■ ' 


X 



Table 4-9. Deslgn^ffects for the new model with t-1 (N-7A4) ' 



£F 


•DEFT 


D2 


D3 


D4 


4.25 


.88 


1.02 


1.10 


1.15 












7.50 


1.51 


2.16 


3.45^* 


3.88 


"'10.18 • 


• -1.92 


3.08 


■ 3.11 


3.51 


15.00 


2.29^ 


4.25_ 


4.30 . 


4.94 


20.00 


2.l0 


3.14- ' 


3.27 


3.42 


26.00 


2.41 


3.87 


3.98 


4.16 


32.00* 


2.46 


3.00* 


3.06 


3.35 


38.18 


2.65 


3.15 


3.27 


3.67 


40.32 


2.66 


3.18 


3.29 


3. '55 


47.50 


2.63 


3.12 


3". 23 " 


4.19 


55.00 ' 


2.57 


3.73 • 


3.89 


4.07 


62.43 


2.55 


3,. 39 


3.63 


3.82 . 


69.04 


2.40 


3.50 


3.49 


3.73 


72.95 


2.34/ 




3.21 


3.48' 


77.51 . 


2.42 \ 


2I84 - 


^.94 * 


3.24 


82.00 


2.45 


3.07 


3. -42 


3.79 


87.47 


2.56 


2.44 


2 78 




91.69 


^.99 - 


1.94- 


2. -06 


2.37 


X 93.53/ 


3.32 


1.87 


2.15 


2.77^ 


96.00 


4.51 


1.41 


' 1.53 


2.29 
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Table 4-10> " Avegoge ^ porfoig taDce of variance estimates , 
" under the new-nodcrwlth t"2 





' ^2 . 

It 


V3 


V4 


Rel-Blas (2) 


29.41 * 


29.72 


35. D2 


Rel-Error (%) 


67.164 , 


r 66.04 


70:59 


Stability 


8.40 - 
77 


9.68 ' 


U.21 




74 ' 


1 

• 





Table 4-11. Average per£oraan.ce of variance ^scisaces 
• under the new nodel with t"l 



■ V2\ V3 • W — 

Rel-3ias (%) 37.69 44.70 ' ^5.08 

Sel-Error (%) ' '68,05 ^ 71.28 79.33 

/ ^t^biJity 10.46 ' .'" . 12.75 15.32 * 

The histo^rans ia fables 4-12 and 4-13 are averaged over nine data sets 
for the first two oodels (Mpntrcal and nev t«2) . The third hlstogran in 
bath tables is based on 20 data sets. The norsal or Gausian-type distri- 
butions agree rather v^ll, at least over the larger symetric inteqfrals with 
thfi, standard normal. Curiously ^ough^ the I-like distributions based 'on 
V2 ha^e*slightly fatter tails for the new nodel runs, while at the saae tioe 
, the stability neasures for 72 increase xrozi for the liontreal csodel to 

8.40 and 10.46 for the new codel. Thes;^ stability ceasures are still 
r. considerable anderestiaates when viewed as appro^cinaticns for tr.e degrees 
of freedom associated with t3ie corresponding I*like distributions in table 
4-13. _ ' * ' ' . / 

Conparison of Taylor Series and Jackknife Linearizations 

In chapter. 2 the Taylor series (TS) variance appro>J^cion icr^^ula 
for a *iAEP P-value equation (2.20) was presented. ^ shown that-- for 
a deeply stratified sanple uith two prinary selections per strattm, thfe 
TS variance estlisator was smaller than the corresponding jackknife (JK) 
^variance approxination. To compare the TS linearization our jack^cnifed 
variance estinators In the context of the slztulation study presented in 
( the^previouis sections, a Taylotized deviation ^ 

■ z^j,- 31(y^j^ - P =^^j/M ' W.44a) 



4 



Pfro port Ion w/ln 
Gaussian 
^Montreal Model 
Hew Model: t-2 
Hew Model: t-1 ' 



Tablg 4-;12. Sampling distributions for Gauslan type statistics 



±2.57£ ±1.960* ±1.645 
.9900 ,9500 .9000 
.9605 ^ .9246 
/^572 .9173 
.9612' .9188 




.9906 
.9916 
.9915 



.282 ±1.036 

.8000 .7500 

.8423 ' .7509 

.8307 ' .7406 

.8326 . .7386 



(-1.^0,0) 
.4750 
.4756 
.4672 
.4774 



(0,1.960) 
.4750 
.4849 
.4900 
.4838 



(-1.036,0) 
.3?50 
.3702 
.3564 " 
.3662 



(0,1.036) 
.3750 
• .3807 
.3841 
<^ .3725 



-J 



Table 4-13. Sawpllng distributions for students^ T-llke statistics tjslng V„ 

. * 



Proportion w/ln 


±2.576.. 


* 

±1.960 


±1.645 


±1.282 


±1.036 


(71.960,0) 


(0,1.960) 


(-1.036,0) 


(a,i.03» 


Student's T(30df) 


. .9848 


.?407 


■ .8896 


.7903 • 


.6915 


.4703 


.4703 


' .3458 


.3458 

• 


Montreal Model ^ 
Hew Model: 't-2 

* 


.9816 
.9792 


.9413 
.9358 


*8967 

;^878 


.8074 
.8024. 


.7158 
.7100 


..4587 
.4583 


.»4827 
^ .4774 *. 


.3473 
.3510- 


.3684 . 
. .3590 


. Hew Model: t-1 


.5787- 


..93M 

; f 


.8958 


.8054 


.7146 


.4644 

• 


i. .4754 


^3563 , 

i 

* i# . 


.3^&4 ' . 


* 








■ 




■ 


% 

* SO* 




1 ■ 
* \ 
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was used in place of the jackknlfe pseudo value 



when forming the estimators V2^ V3 and V4 as specified in equatxone '4.38, 
' 4.39,^ and 4.40 respectively. - . V ^ 

Table 4*14 shows Che results from an Independent s^t of 1,000 samples 
.generated according to thi^ previously described simulation nodel with t*^!. 
Contrasting the first two columns' of table 4-14 with the corresponding 
columns of table 4^8 show^ good agreement between the two independent ^ 
(1»000 samples) replicates* The Daximum standard deviation for the sampling 
e:^pectation of F is .05 percent with an aver£ige across the 20 exercises of 
.01 percent. The simulation estimates of VP, the sanpling variance of P, 
exhibit a maximum standard ^evj^ation of ,12 percent and an average deviation, 
across e:cercises of .04 percent. While it would have been desirable to make 
the Taylor series versus jackknlfe comparison on the same sec of 1,000 
samples, our software .design was such that was more economical to make, 
independent runs than to Incorporate both calculations in the same run. « 

Table 4-15 presents the average performance of the three Taylor series 
variance estimators. Compared with the jackknlfe resJlts m cable 4-11, 
we see a reduction of 3.3 tfo 4.3 percent in rel^Ltive bias. The corresponding 
reductions in relative *total error range from 4.2 percent to 8'.3 percent,. 
The stability measures for the three Taylor series estim^Rrs s^row a 
general fncrease of one unit over the- corresponding jackknlfe estimators- 
This indication of a slight increase in stability for the TS estinacors 
does not show up in the T-like distributions presented in table 4-16. 
While the tails of the TS distributions are more symmetrical than their 
JK counterparts, they are also fatter. The percentage of statistics out- , 
sid,e the ^ 1.960 interval is about 1 percent greater for the IS statiscics- 

Uhile some consideration was given to developing a bias-correction 
factor for. the V2 estimator base*d on these simulation results, it was ^ 
felt that this would-be necessary only #f actual Year 02 sampling errors 
based on V2 were "considerably larger ,than Year Cl sampling errors, ^nil^e 



Table 4-14. Bias comparisons foe the new model with t«l 
and a Taylor-series linea;:i2ation 



*2P • 




^ ET2 


ET3- 


ET4 


4.25 . 


. 0.48 


0.56 


• 

'0.60' 


0'.64 

* 


7.50 > . 


1.43 


1.97 


, 3.11 


'3.49 


iO.'l'A 


2.34 4$ 

3.90. ' 


'3.69 . 


3.75 ' 


- 4.22 


15.00 


■7.11 ■ 


7.16. 


8.25 


*20.00 


4.57 


- ' 6.57.' * ■ 


6.78 


7.13 


26*00 


.6.24' 


9.51. ;~ 


9.87 


10.46 


A2.IOO 


•7.36 . 


8.30 


8.57 


9.39 


•3&.17 


8.44 


9.68 


10.05 


11.26 


4a. 30 


8.73 

1 


10.07 


10.38 


11.26 


-47,50 


. 8.92 - 


10.05 ' 


10.52 
. • .1 


* 13.66 


55.00 . 


8.56 


12.24 


12.72 


15.26 


62.41 C 
69.01* 


8.04 ^ 
*6.85- 


10.57 1 
9.75 

* 


11.32 
9.68 


11.85 
10.41 


72.88 


6.36 • 


'8^51 


8.28 


9.D1 


77.50* 


5.50 


6,60 


6.72 * 


7.43 


82.00 • 


4.79 


5.89 


6.60 


7.33 


.87.50 


*3.81' 


• 3.51 * 


3.98 


4.51 


■91.68 


'3*10 


1.95 


2.08 < 


'l.kl 


93.50 


•2.74 ' 


1.53* 


,.1.76 


r 2.26^ 


96.00 " 


2.31 


0.69 ■ 


- 0.74- 


1.16 






• 







1 

/Table 4-15. Average performance of Taylir-series 
estimates under th& new mgdel with t«l 

r ^ — " — 

. ■ . » -'."'^12' T3 T4 

Rel*-Blas tX)"^ - f ' 34.42 / 46.66 50.58 

Rel-Error (%) ' 63.44 » , 66.48 • 71.02, 

Stability 11.23 ' 13.60 ' 16.31 



.IT / 



Table *"ScimpUiig dilstributlons for T-like stntistfcs; 



_^jiroportion w/ln 


*+ 2.576. 


+1.960 . 


a. 645 


H^l.282 


+1.036 


(-J..960-,0) 


<0,r-960) 


(-1.036,^) 


(0,1.036) - 


Stujieut'Is T(30dfV " 


• .9848 


.9407 


.8896 


■^^.7903 


.6915 


.4703* 


• ./!«.03 


* .3458 


.3452 


Taylor-Series 


.9766 


.9282 


.8732 


.7738 . 


.6750 ' 


" .4620 


.4662- 


.3351 


.3399, 


Jackknife ^ ^ 


^97fiJ 


.9398 


.8958 


.8054 


.7146 




.4754 


.3563 


.3584 • 


TayJor-Sei:les'V3 


.9788 


'.9307 


.8770 


.7772* 


.6788 


• -.4626 


A. 4681- 


.3366 


'i>.3422 


Jackknlfe V3 ' 


■ .9802- 


• .9418 - » 


.8979 


.8090 


. 7192 - 


' •^V.650 


.4769 


.3596 


.3602 


TayJ or-Serles 'V4 


.9802 


.933^ 


* .Bsi^ 


.7824 


.6857 


.4652 


.4684 


^ .3403 - 


.345A • 


JackktilCe VA^ 


.9786 

* 




9004 


.8150 

i 

m 

s 


.7237 

'I " ^ 


.4632 


.4786 

w 


. 3613 

j> .* ■ 


.3o2^ , 



'A 



'■:\ 



'9 ^ * » 



53'. 



I 



Che simulation indicates strongly* that siich variance approximations 
considerably overestimate the variability of.^ * controlled selections,* 
the estimated level of preclslon\ould still, be adequate for National 
Assessment reporting purt>08es. In order^to :;esolve this issue, sampling 
errors were calculated for 131 Year 02 national P-values using a version 
of V2 with the squared difference Jackknife linearization employed in 
Y^r 01 and reintroduced in equation (4.37) » Let^ng h » 1(1)4 index 
JIAEP^s four regional strata, this variance estimator took the form 



i 



4^ (2+) ' %Si 



\2 



h**l % — k*fl k'»k+l ^ * 

A r\ 

he 



with 



+ E Z 

h»l ■ c»l u»l 



Z p^^ (u,u+r)/8 , 



r, +1 = 1 
he 



and 



Y +^y^3^(k,k') 

M + Anij^j^(k,k') 



(4.45a) 



.(4.45b) 



■J 



Alllgj^(k.k'). - ^i^r) 



As\n equation 4.37a ^d b, "S^^"*'^ denp'tes summation ov^' State by mijor 

torn: ceils-£ vlth daia from two or more PSUs (n^j^ 5^ 2) . The second ^ 
ter^^dn equation 4.45 .involves successive. squared differences ^ong 
Single PSU , cells jln ma^^or' stratum^c of regioo^h. Recall that the 
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inde:d.ng u « 1(1) Vr., of single PSU cells follows a particular ordering ' 
of the States in region-h based on geographic pro:<inity. 

The 131 resulting variance* estia^tes were sus:^aarized in ter!2S of 
design effects. This analysis paralleled wherever possible cae breakdowns 
presented in the Cnrony et al. sunaary of Year 01 effect^ -The following • 
chapter >de tails this comparison o£ Year 01 and Year 02 sampling errors. 






* Chapter /o>!PARISOM OFVEAR 01 JdiT) YEAR <JSsSA:IPLISG» . 

. . C ■ • • ■ 

Introduction * * • . - 

The • cooputer^sipalation study oi SaOionaA. Assessment's Year 0, 3^:jple 

described in fH^_jjpeceding chapter suggests th^t the standard variance 

approxiaatioxi^^reconsiended for a controlled selection of ^ririary^nits 

seriously undWeStinate the precision of ??AZP P-values. To assess "i^ne over- 

^all impact of Year 02 sample design changes coupled with significant positive 
^ * ' • ' . / 

bias in the associated saaoling errbr estimates, 60 9-year-old and 71 13- 
year-old reading exercises from the Year -02 Assessijent were exaAinea.. 

^itfed sample de$i^n re<il:s ^(DEFFs) for the 131 nai:ional PTj:aiu^s5 were' ^ 
calculaMJSusing the variance estimator described m .equation i.^3» Similar^ 
sets of 131 DEFFs were commuted for ^AEPs four regions, two sex categories, 
and four SOC subpopulations » ■ . ~ ' 

Stem and leaf displai^s of fchese design effect distribj^itions/^ere f omed » 
to facilitate the calculation of median effects and_oth^zi^ata?le perceTitiles 
[ref. 1], Table 5*1 illustrates cile display qz national Resign effects ior 
the two age classes represented. The left most dol^imn Indicates the first 
tvp significant digits of our "national d€^ign eirects., .-^socia^ed tntrc 
digits ar^ aggregated in rfjoining rows. For example, the aggregate tc the 
right of 1.2 represanirs thre.e, nin,e-year-old DEFFs* taking valued 1.26, i,2o, 
and 1.22. XHe third^ cblumn is a running count from ch^ ^and high ends 
of the distribution toward the center. This tally faci*litates location oi 
the median DEFF and the two quartiles^. These are grpuped below eacr* displa:' , 
along with the starred e:ctreme values* In addition to tnese summary per- 
centiles, the disolav provides an accurate vie^^ of the shape .o&f our design 

* • t ' 

effect distributions* 

Comparative Analysis . ^ - 

^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^ • ^ 

The 131 Year 02 de^ign-^f^cts summarized in table 5-L range fror .9A 
to, 3.93 wtth a median , value of'^2.00* This compaiyes fo a fange of .90 to 
10*88 and a pediaS of 2^.38 for the 1A9 Year 01 exercises e^jamined b;- Chrony 
et^l* Taole ^-2 compares*the distribution of Year 01 ani^ftear'C2 national 
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Table 5-1. Stem and leaf display of Year Oi^tional design^ef fects 

/Niaej Thirteens , TotaJL . 

.5 , . . 

• .6 • • . ' , ■ 

•X ^ - • . • 

.8 . ^ 

.9 54 . . 2 45 . '2 

l.*0 635 • 5 * 356 5 

1.1 9 1 3 6 39 7 • 

1.2 662 4 4592 . 10 22^5669 * 14 

1.3 , ' " - • , . 

1.4 66793 .9 • 50 12^ — ^- 0356679 ' ' , 21 
1,5^ 27298 • 14 • ■ 6 - 13 " 226789 • 27, 
X.6 ' 0138. 18 99 ' ^' 15 013899 3;3 
1.7 *" 1651 22 5629, 19. 11255669 • 41 
1*8 8 23 ' . * 0518214 26 0U24583 * 49 
1.9 067778 29 . , . 2738118 33 0112367777888 62 

2.0 04 -2- * 907930052 -9- 00002345799 -11- 

2.1 15951 29 ' 37 — . 299 1135579 ■ ' '58-~ 
.2.2 021 ,24 53 27 01235 51 

2.3 ''73 21. . 47Q804 25 • 003447?8 * 46 
14 7111 19 564 19 114567 _ 38 

2.5 12 16 303 * 16 01235 32 

2.6 3472 14 , 45 « 13 234457 27 

2.7 019 • 10 «. 019 2). 
^ 5 5 13^ 03459 3*^ 

2.9 4. 6 68453^. 7' ^ 344568 - 13 

3.0 4 . . i - / , 4 ' V 

3.1 '■ < \ 

3.2 729 4 8 2 2789 6 
3.i • • > - 

3.4 5 1 • , 5 , 2 

3.5 , * * 

3.6 - ... 
3.7 

3.9 ' * 3 13 1 

* 1.19 * 0.94 - , * 0.94 

Qll,60 ' Q11.76. ' Q11.69 / " ' 

M 2.02 * M 2.00 M 2.00 ' $ 

Q32.52, ' 'Q?2.45, . Q32.47 

3*4^ ^ * 3.93 . * 3'.93 * 
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Table 5-2 


. Distributions 


or 


Year OX and 


02 national DEr 


rS 


i 




- Year 


0l\ 

1 


lear 




. ^ Desien 


Effect 


IJuisbe? 




Per^nt * 


Nuzjber 


Percent 


/' ♦ 
< 


J. 00 






1 


2. • ' 


2 


ItOO " 


1.50 " 


* 




11 


19 


J4 

t 


1.51 - 


2.00 * 


29 




19 


45 


34 


2.01 - 


2.50 


43 


1 


30 . 


34 


26^ ~ . • 


2.51- 


3.00 






21 ' 


24 


18 


3.0i - 


3.50 


o 
0 




/ 5 


t5 


5 

* t 


3.51 - 


4.00 


10 




7 

\ 


1 


• 1 




4.50~ 


~~ : "5 










4.51 - 


5.00 


3 




' 2 

• 




1 


• > 


5.00 


2 




1 








f 












Total 




149 




100% 


131 


100%^ 



Ih'e Year 01 distribution includes 37 individually acnjinistered in-school 

exercises and 16 out-of-school young adult exercises, while the Year 

distribution includes only ^in-schoc^l grou? adninigtered exercises, 'rnile 

» \ — 

this lack of diversitjv ir. the':node of administration aay explaia so2;e of the 
increased stability shown by the Year 02 DErFs, we suspect that reduced 
variability iil the Year 02 weight distrlj>utions had a core pronounced' effect. 
It is also expected that the inclusion of indivi-dually adziinistered e^tercises 
in the Year 02 distribution would enrich \he lowfer^end of the distribution . 
and shift ,the median further below the Year 01 value. 

^kTiile there appeared to be a cendency for Year (il ^n-school DEFrs CO 
decline as the age -of respondents increased, no consistent trend vas cb- 
sen'ed ia the.Year 02 data^ at least, not between the ages of«oine and 
thifftfien. the regional trend observed in the Yea* 01 data is also ocsciirec 
the Year 02 tabulations. Table 5-3 shows .that «hile the su?rena«" zi 
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Table 5-3. COTiparisp: 




ireg^-oxial DEFFs 



Mode of Subject i Nuriber of Median • DEFTs by Region 



Year Age Administration Area 
01 9 Group Writing 

* 13 Group Writing 



E xerci ses ' NE 
24 
5 



02 



9 
13 



Group 
'Group 



Reading 
Reading 



60 
71 



1.J3 
2.0.4" 



SE 


C* 


W ' 


2.93 


2.32 


2.J55 


3.65 


3.50 ' 


2.65 


• 

2.52 


>1.73 


1.71 


1.66 


1.77 


1.75 









. southeastern DEFFs is maintained at age nine, at age thirteen the trend is 
_ reversed wi^ sjmth^Mst^ low and^gortheast high^ As i n Yea r 01 > the re was ... 
little <iifference betw^^en the sexes with nales registering a oedian DEFF 
of 1,72 and females 1*66 • , > , 

^ Chromy et al* also reported a possible tendency for big 4ity and urban 
^fringe areas to yield smaller DEFFs than the more sparsely populatM medii^Cs^ 
cit7 and small place subpojjulaiions . This Teoden^ ih not apparent in the 
Year 02 DEFFs "displayed In t&le 5-4. • ^ 

In summaryTone can state that Year 02 design effects^are somewhat ^ 
smaller and less variable' than the Year 01 effects • While it can be said 
that the Year 02 effects varied by region and SOC^ there were na consistent 
trends. These factor^ interacted in curfbus ways with the^age class effects* 

Conclusions • i . ' " - 

ICooparing the indicated level of precision for Year OL and Year- 02^ 
NAEP exercises, it was apparent t^iat in spite of the suspected positives 
bias in Year 02 sampling errors, the overall level of pfreciSion waS ixroroved 
somewhat in Year 02*' In light of this result, no bias correction <vas 
attempted for. the Year 02 variances* The sampling error approximation which 
uses squared differences of j^cklpife pseudo-values within control cells an4 
squared Successive differ en ces^tween single PSU eel Is, within major size 
of community by'-SES substrata (V2 "was judged to be the legist biased of the 
computationally feasible estimatofs. The jackknife linearization was 



J ^ 
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""aqle 5-4 • .Y^ar^ 02 DEFFs by size of* cozzunity 



Mode of ^uSject Jlimber of f 



Age 


Adninistratioa 


Area 


Exercises. 


B.C. 


U.?. 






• 






9 


Group 


Reading 


60 


2.38 


1.64 


13 

1 


Group ^ 
* 

1 


Reading 


71/ 


-^.05 


1.66 


"Total 


Group 


Reading 
1 


• 

131 


2.16 


1.65 


\ 











>fe<ilan DSFTs bv SOC 
M.C. *S. 




2.05 



2.05 



retained since tne iaprovenent deaons traced for t^ie Isvlor series ^stizators 
did not justify the added cost oz redesigning SAE?s saz:?l:;.n2: error ^oitvare. 

* National Assessments in^schcol and cut^oz^schcol designs for alsess-t 
cent Years .D2^tivrou^ .rgsaineri basically --the sase vitn controlled 
selection used at the prinary scege to allocate ?SZ^^ ftate by Mjor 



stratiia control cells vithin regions/ The sanplin^ error cetho do Ic'gy 

developed for the Tear 02 saaple has be« applied dire.ct^y to calcu^te 

Year 03 and Year 0~ saxoling errors 
WW 



'In viev~or ta€ difficulties associated with producing reasonably^ 
unbi ased sanpling error^estiaates for controlled selections, a cajor re- 
design of XAZ?s prisarj sample was initiated foV the Year 05 assesszent. 
The Year 05 (l^/j^-Ti) primary saaple included the/ 15 largest S::£As 'in 1970 
as se If -reo re seating ?SUs- Saanled fSUs ^'ere stratified bv region, State 
and site of cosiiunitXr XAEP's requirement chat ^11 Stapes- be represented 
in the -sanple wa3 cet by carving but a stratua within eaph State which vas 
not already covered by ^- se If -rep res en ting SMSA« These State strata vera 
yCssigneS two priaary selections wher^vex size ^^x;^tted. Scae single JVC 
strata were carved out in small States* Listing ictiis fr^a 'States alreadv 

covered by self-reo resenting SMSAs and those not contained within the 

'4^ ' ' ' \ ' • 

/'carved out^'-S^tate sujjstrata were placed in. a regional V^i^' The regional 

ypcJol was stratifieli along size of ccraunj^tv lines watr* rvo or three selec- 

tions per stratum* ^ 4 ^ • . i 

Sampled PSUs were selected vith pi^babilities proportional, to tne^/ 

14-year-ol4 population in 1970 with siae adjustcsent to effect an ov^rsaapl:-n 



A i ' ' ' , 

c 

of PSUs cotJtaining low Incoaie inner city areas or highly rural counties. * 

Sanpford's yejective PPS without replagenent selection oethod was used. 

Aside frpffi the, 1^ self -representing SMSAs, NAEP's Year 05 pripary san^jle can 

be descr&ed as a deeply ptratified PPS selection of two or threfe prliaary 

^^^^^ r ' ' . 

units p&r strata. The few singie PSU strata carved out of snail States 

*were paired within regions for \^ariarice' estimation* In order to account 
for the withln-PSU variability of the self-represcnters-, replicated school 
samples were dravn. With the planned collapsing of s:^gle PSU strata and 
the replicated sdtool samples within se^Lf-representers , a variance approx- 
iaatlon based" on squared differences' between expanded-up PSC oontributions 
(or repliqate contributions) within -strata 'st^oul^ be rfascnebly unbiased* * 
Soae ov*resti!aation could be expected due to l^Cjring the effect of without- 
replaceaent selection of PSUs an4 replicates. detailed desctiption of the 
Year 05 NAEP sanples can be foimd in RTI's final report for assessment Year 05 
[ref. 2]. \ ' V ^ > 

The Year 06 (1974-75) NAZP in-school priiaary. sanqple x/as. essentially an 
independent replicate of the Year 05 sanple selected from the ^eply strati- 
fied prinary jmit franq^evelop^d for the-^^^*^^ survey. Variance estiiaates 
for Year 06 statistics were ag^in obta^ed frot^quared differences of PSU 
(or replicate) level Jackkxiife pseudo-values sucnted over prioary strata. 
For the Year 07 HA£? saople, a decision was m&^e to draw four non-overlapping 
sanples to be used successively for Years 07 through 10. This was accom- 
plished by adapting the deeply ^tr^ified Year 05 design strategy to select 
enough PSUs and replicated school sanples within thq self-representing 
primary units to serve for four years. The cotobined sample w^ then parti- 
tioned at random into four e^ual $ized»^ yearly samples. To preserve valid 
PSU replication, prinary strata in the.oaster sample were cos^bined and the 
associated prlssarys were randomly partitioned into four ^ets each containing 
two or^ipccasionaHy three units. By relaxing the all-state requirement to 
assure coc^lete state coverage over, the four year sample, it was possible to 
assure that no school would be visited more than .once during the Year 07 
through Year 10 assessments. . * ' * 



The variance estiraacion aethodology adapce'd\for the \ear 05 and Q|6 
Samples was nodified co'su;! ^jjlTared differences tjfetv^en ?SZ (or replicate) 
psuedo-values and the corresponding prima|y stratum nean. With more 
primary scraca containing three units this modification was nade 
bring IJAZP's variande esrticia't^on in line w^th the general jackknil 
approsiaation ecoixfended, in chapter 2, equation 2.12. A detai] 
description of^ the nonoverlapping Year 07 through Year 10 NAEi 
can be found in RTI's final, report for assessment Year 07 [ze£ J]- 




1. 
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Appendix A: TAYLOR SE?vIZS Lf?EAKmTION FOR REGSESsfOK COEFrlCIS^rrS 

Population Sefinitions " . 

Consider a finite universe C with N units U(i). Associate vith the 

1-th unit a vari'ate val^e Y(i) *and a vector of (p^l) regressors 

" , X(i) - <X-(i) X,(x) *..*X (i) >/ , (A.l) 

0 1 p 

The linear predictionr equation for Y(i) of the fora 

y(i) - X (i) 6 (a: 2) 

vhich iRiniiiizes th€^ svn of squared deviations 

ieU 

is tne fanllxar least-squares regression equation where^B is a soluti&n to 

^' - , b / 

tne so-called *nomal equations' 



where* 



T - ^ (" 



and 'S * • * 

' ^ ^ ' X^Y ^ :: x^(S) Y(i)- , 

isU ■ 

If X*X has rank there is a unique solution for t in (A*A)j nacely - 

If the p^l equations ^represented by th'e natrix equation (A. A); are not linea 
inaependent, redundant equation^ can l>j^ replaced by independent linear ■ 
* i^gstriction^ on the £s* Vnile the following development i^^^Il^iited tO' the 
full rank ca se, it is not difficult t^ e::tend"the results drrectly to a 
particular^ re^ticted solution. . | 



nation 



ERLC 



Suppose that a sample S of n units is' selected froa the universe U. 
Let^7(i) denote the probability that unit l'(i) will be included ir. sucr. a 
5ip?le\ jJnbiased Horvitz-rnoiipson estimators for X^X and X Y are 

z, X'(k) X<k) >-(k) ' ^ (A--&a)' 

9^. }'..'■' I n 94 - ■ . . * 



,and 



Oc'^y) - Z X^(k) Y(k) 77r(k).^ " , {A-6b) 
kcs 



Using these estimators,N«.e solve for b in the estimated set of normal 



equations V . ' ' ^ 

_i , .(x^x)b' - (x^y). (A- 7) 

^ As an ^timator of 6 the population vector of regression coefficients 
T *-l T 

b » (x x) (x y) can be viewed as a matrix version of the combined ratio 
esti^tor R » (X) (Y) . TJiis analogy will be strengthened by die form of 
'the Taylor series approximation for (h-S) • , ' 

Taylor Series Variance Approximation 

To generate the first order Taylor series approximation-- for b, we 

^Ibegin by evaluating the partial "derivatives ^ 
- < ^ • ' . _ 

3b /a (x y) for j - 0, 1, p ' (A. 8a) 

3 • , , ^ 



f 

and > 



fcftr j * 0, 1, p {A-8b) 



T 

wlxere i(x y) . represents the j-th element in the (P+1) x 1 column vector 
3 * , ^ - 

T T 

<x y) and (x x) is th^ (jj')-th element of the (p^l) x (p+1) symmetric 
T 

matrix (x x) . * . - 

First, we notice that " * _ . . . 

' * . {3(x'^x)b/3(x^3j) } - (x'^^x) Ob/S (x'^^y) } , {A.9) 

(x'^x) {3b/3(xVj ^ (A. 10) 



Therefore 



where. 



j Jir otherwise 

• ■. . ' 

This allows us to write 



3^^^''^ . jO , {3b/3(x^y)^} - (j:\)"^6j . (A.ll) 



/J 



> 



^tice tSat is a CP"^1) 1 coltmn vector with a 1- in rov j and zeros ^ ^ 

eibewnere-^ To ev'^luate the partial derivative of b with respect to the 
eleaencs of (X X) we note that. 

•3 (x^:0'b/3 (x"x),^,:- - S (x^O/S -(x'x)^^, - 0 (A. 12) 

ence ■ u ' „ • * 

(:c"x) Ob/3(x\)^j-,> + {3(x'x)/5(x^:0^^,} b - Q ^ (A. 13). 

Recalling that (X X) is syinnetric, we see t^iat ' \ 

' \ * 

^ • {3(x\)/3(x\)j^,} - D^^, ' ■ • ; * (A. 14), 




where the (r,c)-th kleaient o£ D,,,'is • 
d,,^,(rc) * [1 - c(jj')]:5,(r)o (c) -r 5,,(r)o~ (c) 
♦.iith = i if j ".j' and zero otherwise. This leads to 

'^where . , is a x (p?l>--£Kiaetric nacrii: having Is in oositions i.j j ' ) 

. 1 _Zr^= — \^ 

an(i (j'j) zeros el£et?here. 

EvalUjpcmg the paKial derivatives in equations (A* II) and (A. 15) at ihe 
point <('x :0 (:< y) » X Y > ve can "appr Donate b '^•ith the firs^;^ orcar 

Taylor se;ries linearization 



^ ' • ' ■ : JUV). - (X^Y )]{3b/(x^y;>.> 



b = fi^ : J(x^y) - (x^Yj]{3b/(x^y;i.> ' u <Aa6) . . 



^ z : [(x'x),,,';- (x':{).,,l 'ObynxM,,,} 

which becomes p ^ ^ * . . ~ 

b i 3 .+ (K^X)"^ .Z'^ (-(x^y), 7 (X^Y),] c, 



P ? 
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(A.r) 




Recalling the definitions of 5.- and D.., it is easy to s6e'that . 

' . ' 2 ■' ( (x'^y).^. - (x'^Y) J 6, - (' Cx^y) - (x'^Y) ] (A. 18a) 

and „ « ' . ^ 



« T 



2 2 [(x^x),,. - (X^X).,.l D.;, • [(xM - (X^X)]\ (A. 18b) 
This allows us td rewrite (^A,17) as . f ^ 

^ " b,i & + (X^)"^ {[(x^y) - (X^)]'- [(x*^*) - (Vx)]6}.^ '(A. 19) 
Finally, observing- that (X^X)6 .- (X^Y) have 

b i 8 + (X^X)*^ [ (x^y) - (x^x)6], \ ■ (;^.20) 

To exploit thfe result in (A.20) in order tb\ approximate the Banpling 
variance of we can define »^ 

\ 

3(k) - (Ji^X)"^ [ '(x^y)j^ - (x'^x)j^6] (A.21) 

where (xV)!, - x'^(k) Y(k) • , *' . ^ 

- ' , », „ 

.Ck x). - X^(k) X(k)»- 

- 

The corresponding Horvitz-Thompson estimator^li • , , ' 

#3-2 3(k)A(k) . /• 

' keA . . 

• - (X'^X)"^ t(x^y) - (x\)6]. • ' ' U.22) 

■ '} . : % ' ' . 

CombinJjQg CA.^O) and (A. 22) we see thait 

n * " - " 

' . r 2 - (b- 6). ' \ ' ' (A,23) " 

The result in (A. 22) leads one to ^fic following approxijnation*for the 
generalized mean-squared-error of b^ our vector of p + 1 estimated 
regression Coefficients t * • - * 

GMSE {b'} -'e {(b-gXb-B^'^} • • ' 

8 

I 

* VAR {2} • > ' ' 



5ince E -S) ^ 0, the x 1 null vector* 

If VAxL. (T) is the saapling ^variance formula for an estir.ated total T 
f rota a particular ganjplins design (D) , then the generalised nean-squared- 
^ error for^~ve^o~r^o'£ (pH^l) regression coefficients, b est^nated fron D is 
^'approxiniately • " . - J* 

-where S,is the linearized statistic* 

. In the following^ section, we will exploit equation (A. 25*) to, produce the 
'laylor Series' variance estinator o^nore precisely, nhe generalized Man- 
squared-error estimator' for b. ' / 

Taylor Series Variance Estimator 

Recalling the Taylor series approxiraaciton f^or GISE^{b< develoded in 

the previous section, it is^ clear that ii var^^CT) represents an^appr^?ria:e. 

variance estLr-ator for the sample total T frosi design D, then var-(S) is 
• * * )j \ 

th^ assocy.ated estinator lor^GIISE {b^* Since the linearized variate value 
/ ^ * ^ T -1 

(k) m (A. 21) is a fcunctioi) *Df the unknown population --quantities (i; \V) 

. and £, cijae*^ obliged to inpose another level^of approxination at this 

point* Instead 5(k) , we use ^ ' 

^ z(k) - (x^x)'^ { (xV;; - (=^Vk ' ^-^^'^^ 

subs ti tut ir.g jZ'ur saniple estiinates for the unknown^ population parameters. 
It is interesting to note at this"" point that (A. 26)^ is a matrix analogue or 
the "Taylorized*. variate used to approxinate the variance of the ratio 
R ■ E:q)ressing this ratio as R ■ (X)"^(Y) and making the associations 

(X)'^ <-> (x^x>"^ and (Y) <-> (x^x) . -the relationship ^betwLn (A. 26) ^nd 



the familiar 



is obvious. • 



2s(k) - (Y(k) r R X (k)J / X • 

, ,- (X)"^ tY(k) - X(k) \' (A. 27) 



To illustrate the roethod, ^e can corts'ider a stratified s'ifcle randi;n 
Cluster sanple with h''l(l)ii strata, and 131(h) clusters selected ^rora 



* - t 
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withbut ^replacement. If y(hik) Is the varla):e value assoclat&d with the 
k-th unit t}c"l(l) M(hi)l ift cluster-i of 8tratum-h and X(hik) is the- 
coijresp ending 1 x tp+D row' vector , of regressors , 'we us'e the expanded 
ctuster totals ♦ . # * 



(x y), , - N(h) 'I X^(hlk) Y(hik)/n(h) 
h-1 



(A. 28a) . 

V 



and 



to form 



_ . . "M(hi) 
(xV)t,. - 2 X^(hik) X (hik)/n(h) 

, k-1 



^ ^tihi) - [ (x'y)., - (x"x)^v b) 



T 



hi 



'hi 



(A. 28b) ' 



(A.29) 



wherQ 



and 



b » <x^x)**^ (x^y) 



(x x) -IS 



("SC x), . 

h-1 i-1 



(x^y) 



H n(h) _ 
I 2 (xy) 
h-1 i-1 



hi 



Then, we calculate 



gfase {b} - (x^x)'^ { Z ',{r- f(h)} nd) s^ (h)} (x^x) 



H 



•v T ,-1 



where f(h) - n(h)/JJ(h> and 
2 > n(h) 



z 



h-1 



I . t2(hi) - 2(h.)ll2(hi) - 2(h.>r/[n(h)-lJ 



(A. 30) 



i-1 

n(h) 



■witfi-2(h.) - Z 2(hi)/n(h). 
i-1 

Statistical: Inference 



Researchers are often intereste*d in, testing hypotheses about reiatidh 



ships amon^ population regression coefficients* While there is no rigorous 
solution to the general linear hypothesis problem in a finite population 
context, we suggest )i heuristic approach ^lich relies on the ceptral limiting 



tendenc/ .o£ estimated «eff IcUnt ,„i.nce-covafiil.« 

.a«ix,^..„ji5 as. specified in • - ; ,,rfoll<«inS 

ti„nfor*es»:»plinsdfstrib«io„oH.l>o.ds..on :i 

appwaeh for c«ting ' Z ' ' ^- , ., " (a.M)", 

- o versus t^/^- ^ 

Che test s„tisuc . ,.1,-T,;„ ' (A:32) 

. , , « H if -ee.s the upper a peilenta^. point of th, Chi-S,»are 
teaect H„ if ^^^^^^ ^^^^ 

6istri*otion «ith c - rank.(0) deg. 

— enUo.e of - — --^C^^^^^^^ .he estimate, variance- 

covariance «tti. var„(« a P „„,,iii„,.s statistic. The- . 

■ variate analogue .f Stc ent s T n» 1 . ^ . 

r-transfoi==a version ol Kotelling s is . . 

, - •» , if * 1- c , . " 
- r - t- (di)c ^ ^ • <: , 

1 

■ • "of fi«doo associated with v« 0). Tor our 

„i,ere df is the desrees of the .approxination 

, e,a.ple in the previous section « - ^„ ,,„pje ind. ' 

o/4.^ 4*: -the tbtal nunber o* cxusuk 
df * n(+)-H where n.+) is the ; ,,,,.j,j^c t is compared to the 

- H is the nuniber of strata.. The transformed . (df + 1 - c) 

. ^ - * T4<iher's F distribution with ca^.v." 
upper a percentage, point of risher s * . . 

degrees of freedon. • . ~ . * " 



f 



■ • * 
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